Active Record Query Interface 3.0 126
I’ve been working on revamping the Active Record query interface for the last few weeks ( while taking some time off in India from consulting work, before joining 37signals ), building on top of Emilio’s GSOC project of integratingARel and ActiveRecord. So here’s an overview of how things are going to work in Rails 3.
What’s going to be deprecated in Rails 3.1 ?
These deprecations will be effective in Rails’ 3.1 release ( NOT Rails 3 ) and will be fully removed in Rails 3.2, though there will be an official plugin to continue supporting them. Consider this an advance warning as it involves changing a lot of code.
In short, passing options hash containing :conditions, :include, :joins, :limit, :offset, :order, :select, :readonly, :group, :having, :from, :lock to any of the ActiveRecordprovided class methods, is now deprecated.
Going into details, currently ActiveRecord provides the following finder methods :
- find(id_or_array_of_ids, options)
- find(:first, options)
- find(:all, options)
- first(options)
- all(options)
- update_all(updates, conditions, options)
And the following calculation methods :
- count(column, options)
- average(column, options)
- minimum(column, options)
- maximum(column, options)
- sum(column, options)
- calculate(operation, column, options)
Starting with Rails 3, supplying any option to the methods above will be deprecated. Support for supplying options will be removed from Rails 3.2. Moreover, find(:first) and find(:all) ( without any options ) are also being deprecated in favour of first and all. A tiny little exception here is that count() will still accept a:distinct option.
The following shows a few example of the deprecated usages :
1 2 3 4 5 | User.find(:all, :limit => 1) User.find(:all) User.find(:first) User.first(:conditions => {:name => 'lifo'}) User.all(:joins => :items) |
But the following is NOT deprecated :
1 2 3 | User.find(1) User.find(1,2,3) User.find_by_name('lifo') |
Additionally, supplying options hash to named_scope is also deprecated :
1 2 | named_scope :red, :conditions => { :colour => 'red' } named_scope :red, lambda {|colour| {:conditions => { :colour => colour }} } |
Supplying options hash to with_scope, with_exclusive_scope and default_scope has also been deprecated :
1 2 3 | with_scope(:find => {:conditions => {:name => 'lifo'}) { ... } with_exclusive_scope(:find => {:limit =>1}) { ... } default_scope :order => "id DESC" |
Dynamic scoped_by_ are also going to be deprecated :
1 2 | red_items = Item.scoped_by_colour('red') red_old_items = Item.scoped_by_colour_and_age('red', 2) |
New API
ActiveRecord in Rails 3 will have the following new finder methods.
- where (:conditions)
- having (:conditions)
- select
- group
- order
- limit
- offset
- joins
- includes (:include)
- lock
- readonly
- from
1 Value in the bracket ( if different ) indicates the previous equivalent finder option.
Chainability
All of the above methods returns a Relation. Conceptually, a relation is very similar to an anonymous named scope. All these methods are defined on the Relation object as well, making it possible to chain them.
1 2 | lifo = User.where(:name => 'lifo') new_users = User.order('users.id DESC').limit(20).includes(:items) |
You could also apply more finders to the existing relations :
1 2 | cars = Car.where(:colour => 'black') rich_ppls_cars = cars.order('cars.price DESC').limit(10) |
Quacks like a Model
A relation quacks just like a model when it comes to the primary CRUD methods. You could call any of the following methods on a relation :
- new(attributes)
- create(attributes)
- create!(attributes)
- find(id_or_array)
- destroy(id_or_array)
- destroy_all
- delete(id_or_array)
- delete_all
- update(ids, updates)
- update_all(updates)
- exists?
So the following code examples work as expected :
1 2 3 4 5 6 7 8 | red_items = Item.where(:colour => 'red') red_items.find(1) item = red_items.new item.colour #=> 'red' red_items.exists? #=> true red_items.update_all :colour => 'black' red_items.exists? #=> false |
Note that calling any of the update or delete/destroy methods would reset the relation, i.e delete the cached records used for optimizing methods like relation.size.
Lazy Loading
As it might be clear from the examples above, relations are loaded lazily – i.e you call an enumerable method on them. This is very similar to how associations and named_scopes already work.
1 2 | cars = Car.where(:colour => 'black') # No Query cars.each {|c| puts c.name } # Fires "select * from cars where ..." |
This is very useful along side fragment caching. So in your controller action, you could just do :
1 2 3 | def index @recent_items = Item.limit(10).order('created_at DESC') end |
And in your view :
1 2 3 4 5 | <% cache('recent_items') do %> <% @recent_items.each do |item| %> ... <% end %> <% end %> |
In the above example, @recent_items are loaded on @recent_items.each call from the view. As the controller doesn’t actually fire any query, fragment caching becomes more effective without requiring any special work arounds.
Force loading – all, first & last
For the times you don’t need lazy loading, you could just call all on the relation :
cars = Car.where(:colour => 'black').all |
It’s important to note that all returns an Array and not a Relation. This is similar to how things work in Rails 2.3 with named_scopes and associations.
Similarly, first and last will always return an ActiveRecord object ( or nil ).
1 2 3 | cars = Car.order('created_at ASC') oldest_car = cars.first newest_car = cars.last |
named_scope -> scopes
Using the method named_scope is deprecated in Rails 3.0. But the only change you’ll need to make is to remove the “named_” part. Supplying finder options hash will be deprecated in Rails 3.1.
named_scope have now been renamed to just scope.
So a definition like :
1 2 3 4 | class Item named_scope :red, :conditions => { :colour => 'red' } named_scope :since, lambda {|time| {:conditions => ["created_at > ?", time] }} end |
Now becomes :
1 2 3 4 | class Item scope :red, :conditions => { :colour => 'red' } scope :since, lambda {|time| {:conditions => ["created_at > ?", time] }} end |
However, as using options hash is going to be deprecated in 3.1, you should write it using the new finder methods :
1 2 3 4 | class Item scope :red, where(:colour => 'red') scope :since, lambda {|time| where("created_at > ?", time) } end |
Internally, named scopes are built on top of Relation, making it very easy to mix and match them with the finder methods :
1 2 3 | red_items = Item.red available_red_items = red_items.where("quantity > ?", 0) old_red_items = Item.red.since(10.days.ago) |
Model.scoped
If you want to build a complex relation/query, starting with a blank relation, Model.scoped is what you would use.
1 2 3 | cars = Car.scoped rich_ppls_cars = cars.order('cars.price DESC').limit(10) white_cars = cars.where(:colour => 'red') |
Speaking of internals, ActiveRecord::Base has the following delegations :
1 2 3 | delegate :find, :first, :last, :all, :destroy, :destroy_all, :exists?, :delete, :delete_all, :update, :update_all, :to => :scoped delegate :select, :group, :order, :limit, :joins, :where, :preload, :eager_load, :includes, :from, :lock, :readonly, :having, :to => :scoped delegate :count, :average, :minimum, :maximum, :sum, :calculate, :to => :scoped |
The above might give you a better insight on how ActiveRecord is doing things internally. Additionally, dynamic finder methods find_by_name, find_all_by_name_and_colour etc. are also delegated to Relation.
with_scope and with_exclusive_scope
with_scope and with_exclusive_scope are now implemented on top of Relation as well. Making it possible to use any relation with them :
1 2 3 | with_scope(where(:name => 'lifo')) do ... end |
Or even use a named scope :
1 2 3 | with_exclusive_scope(Item.red) do ... end |
That’s all. Please open a lighthouse ticket if you find a bug or have a patch for an improvement!
UPDATE 1 : Added information about deprecating scoped_by_ dynamic methods.
UPDATE 2 : Added information about deprecating default_scope with finder options.
Comments
Sponsored by
About
I'm Pratik Naik - A Ruby on Rails developer, an Indian expat living in Cambridge, UK and working for 37signals.
Currently a member of the Ruby on Rails Core Team
pratiknaik (at) gmail (dot) com
Thanks for all your work on this! I can’t wait to start using it.
One note, looks like the with_scope code snippet is missing a closing parenthesis.
@Ryan : Fixed that, thanks for pointing it out!
Nice! I really like these changes. Coming from Django to Rails, I miss Django’s database API which support chained methods (among other things). Having an API like this is easy to remember and work with.
Django’s filtering gives you a way to traverse relationships, for example `posts.filter(blogname=”My Blog”)`. Is there a similar way to do this with the new AR interface? Something like `Post.where(:blogname=”My Blog”)` would be handy!
@Kyle : I’m not sure I follow. Why doesn’t Post.where(:blogname => ‘MyBlog’) work for you ?
I’d love to see how and when the queries are actually executed under this new paradigm.
Awesome. I can’t wait. One picky correction. I assume:
is in correct and could also return nil. The new scoping is beautiful. Any chance we’ll be able to use complex scopes to scope associations?
I share Ryan’s sentiments – thank you, this looks great, can’t wait to start using it.
This is all looks like a massive step forward. One question – why isn’t this being deprecated as of Rails 3.0? I agree with waiting a couple of releases before switching the old stuff off, but the sooner deprecation warnings appear, the sooner people will step up and fix them.
Pratik, this looks great!
Strangely enough, I found myself beginning to use named_scope to create some of the new API methods already. I think this will be a great improvement over the current way of doing things.
Great job, looking forward to seeing this incorporated into Rails!
Solid implementation. The Rails pipe dream I’ve always had finally come true :p
Does this mean that we can use scopes from other scopes now? For example, does this work?
class Item scope :red, where(:colour => ‘red’) scope :since, lambda {|time| where(“created_at > ?”, time) } scope :recent_red, lambda { red.since(1.day.ago) } end
Sorry, I might be slow but it just looks looks like change for the sake of change to me.
I can see a couple of uses for the chainable filters, but not enough to overhaul the entire API. How is this better?
That is bags full of awesome mate, congrats to everyone involved :) Will be working on refitting an app currently on iteration 1 this weekend, will let you know how I get on :)
so suppose we have the relationship “author has_many books”. how would i write a query like “find all the books written by author X”
Bravo, Pratik. Really good work.
@Seth, this is definitely a huge improvement to Active Record. Although I haven’t taken a look at the internal code, previously there was a lot of duplication in how finds, associations, and named scopes handled things. I’m guessing this improves the internals significantly. Everything goes through the same Relation logic.
Not only that, but it brings a lot of power up to you like lazy loading.
@Pratik, does a has_many association go through Relation as well? Such as “post.comments”.
Hi!
Would be nice, if you would add the fired SQL-Statements as a comment to the examples above. Just to see, which statement is fired when.
I really do look forwards to Rails 3. Thanks for the great work, guys.
Bravo, Emilio and Pratik for all the work on this. I’m curious about what kind of support you’ll offer for plugins that want to modify ActiveRecord’s behavior; the recent changes pretty much completely broke my friendly_id plugin because I was relying on ActiveRecord::Base::find_one and find_some, where are now refactored away. It would be great to have an “official” API that plugin authors can program against to avoid having to relay too much on hidden internal stuff.
While I really dig the new syntax, isn’t it going to be way harder to build search interfaces? Instead of passing option hashes to methods, now we have to chain some methods, even calling some conditionally? I’m trying to understand how one would call .order conditionally based on a param prior to fetching records.
I do not like this.
Building queries will get harder and reuse through methods that just build some hashes will be limited, you can count on me building a plugin that will reverse this change/deprecation.
Building a hash/reusing/merging it with others is a fundamental way of building options.
This looks pretty awesome so far. Is there a plan of making using multiple databases easier for things like data migration? It’s a really useful feature in DM and would be nice to get in AR.
I don’t understand why we have to use lambda to call named_scope. It’s a very clunky API compared to passing a block. I think we should optimize the common case. I understand that’s hard to deprecate, but it might be detected based on the block’s return value…
Interesting. Congrats on the effort.
When will this be available for some testing (not Rails, just the AR gem).
Will it be possible to use selects with include now? Right now in rails 2.x if i want to use eager loading it’ll overwrite any select values I pass in to active record. Unfortunately it means that eager loading is only really useful for very simple scenarios.
Awesome! I love that the core team isn’t afraid to completely change the API in favor of progression. Keep up the good work!
This seems pretty interesting. Going to take awhile to think this through.
This seems awesome: “A relation quacks just like a model when it comes to the primary CRUD methods. You could call any of the following methods on a relation”
grosser’s comment seems pretty true to me, but it seems like there might be a way around most cases I’ve done. For instance, if I’m incrementally building a set of conditions, I still could do that, I’d just pass those conditions to a where instead of a find. The problem would be if I want to throw conditions and an order in at the same time. I believe then you’d have to keep the options together than pull them apart or something..
It seems like it’d be incredibly handy to have an add_relations method that you could pass the older style hashes into, something like:
User.add_relation(:conditions => { :login => ‘hi’ }, :order => ‘created_at’).other_scope
Also, if these scopes are getting a .new method, seems like that should be synched up with relationships, and they shouldn’t have a .build, but should have a .new.
This is all good, but how are gems and plugins going to be backwards compatible? Or are we expected to have the “rails 3 or nothing” mentality?
@Pratik: Sorry, to clarify, assume that `blog` is an association. Django allows you to traverse relationships with a double-underscore syntax .
(It appears the double-underscore broke the comment formatting. This better illustrates what I mean: http://gist.github.com/284020)
@grosser, this may be just my opinion, but merging nested hashes gets messy very quickly and is one reason I’m so excited about this new API. Especially when conditions/order options are strings and need to be merged with delimiters.
As for re-using hashes, where would you do this where you can’t use a Relation? The with_scope method is a perfect example of passing a Relation into a method where a conditions hash would have been.
If you have a code example showing where a hash works better than this new API, that would help.
Looks good Pratik. So, with the lazy loading, would that mean that:
Would just be one query with a join, instead of 2 queries as it seems to be in Rails 2.x?
Kyle Fox: You'd do something like @posts.joins(:blog).where("blog.name" => "My Blog")(I haven’t tested it, but I’m guessing that’s how the :joins option works now.@Richie Vos and grosser: I don’t understand what you’re saying. If you want to pass conditions and order at the same time, you just call model.where(some_conditions).order(blah)
@Chris Eppstein: +1, completely agree with that.
It’s awesome to see all these in. For once associations, scopes, and normal finds will just behave alike :D
Who is awesome? You are awesome! Nowadays the only thing I missing in AR – Class Table Inheritance.
http://martinfowler.com/eaaCatalog/classTableInheritance.html
@Donnie : The queries are executed when you call .all or any of the enum method ( each, each_with_index etc. ) on the relation.
@Steve : Fixed that, thanks!
@Tom : I didn’t want this to be major blocked in upgrading to Rails 3. In my experience, people really freak out on seeing deprecation warnings and consider the functionality as good as gone. Considering this is going to deprecate almost every Rails app ever written, I think the advance warning is fair.
@Joe : Yup, that’d “just work”.
@Seth : Keeping aside internal cleanups, it’s much less typing and is much more succinct. And makes it a lot easier to do optimization work if/where needed.
@Ryan : Even though all these finder methods are available to associations, associations don’t go through Relation. In theory, it should be possible to build associations on top of Relation. Not sure how much work would that be though.
@Norman : Yeah indeed. Relation has many more methods than I have blogged here. I’m sure we’ll have a decent Plugin API finalized for this before the final release. In the meantime, you could go through the code and make suggestions about what you would want in the plugin API.
@grosser, @Wynn : I’m not sure I follow. If anything, this should make conditionally building queries a lot easier, as the query doesn’t get fired until you can an enumerable. If you have an example, I can explain it further.
@Lucas : You could just clone Rails and have a go – http://weblog.rubyonrails.org/2010/1/1/getting-a-new-app-running-on-edge
@Richie : Relation already has a method called apply_finder_options, which is internally used to support the old finder options. It’s just 4 lines really.
very nice new API… unfortunately I think it will break a lot of older rails apps…
Wouldn’t it have made more sense to just let rails 3 support AR or Datamapper?
@Richie I prefer .new over .build really, but relation#build is already aliased to relation#new for consistency.
@Ben : The existing syntax is going to work even in Rails 3.1. I’m pretty sure you don’t support Rails 1.2 at this point ! Even then, converting old finder options to a relation is a tiny method
This looks like a huge amount of progress. I remember when Arel was just a gleam in Nick Kallen’s eye. Nice to see it finally get real.
What about find_all_by_x and find_by_x! finders?
@Josh : find_by_* and find_all_by_* methods are all available on Relation. I don’t think they’re gonna be deprecated, at least not in 3.1. However, scoped_by_* methods will be deprecated.
It seems that named_scopes or scopes should be deprecated entirely if the new finder methods chain properly. They are no longer necessary, instead of creating a named_scope, you could simply create a method that does the same thing. This is also assuming that your chain will check methods on the class(model).. if they don’t, they should. That way you could eliminate named_scopes entirely.
Quinn, we considered this, but the def self.scope/end noise for something so common was deemed unhelpful. This also makes the purpose very clear. You can, however, choose to just use methods if you please, but the recommended way is to use scope.
white_cars = cars.where(:colour => ‘red’)
I guess that should be “red_cars”? :)
Looks nice. How should one write Rails 2.3 apps now to be prepared for this?
@gregp: Rails 3 DOES support DataMapper, but this is a pretty nice addition for those of us with a substantial investment in AR already.
Some excellent work by all involved.
One query though: Are relations lazy loaded in non-enumerator situations?
The case I’m thinking of is a show page for example:
Is @post a relation object, or the Post object at this point?
If it’s a multiple relation object, would someone need to run @post.first on it before using methods?
Or is it a single relation object, in which case @post.title would work?
So in the case of caching in the view, where would the user expect to have the query fired? In the controller with .find() or in the view when they first access a value of the object?
The ability to have a lazy loaded find(), which is only loaded if one of the cache block needs it would be very powerful. That way, if all caches were filled, no SQL query at all! And no extra work on testing all caches exist to get that functionality!
Looks like a great overhaul of AR! Keep up the good work guys!
@grosser, @gregp, @Ben
As Lifo pointed out, there is a method in AR that can convert the old hash you passed to find(). But with very little code, you can write your own one to suit. The following should be possible: http://pastie.org/790395 (untested)
this, is, awesome!
I would like to see some examples of more complicated queries. While I think these changes look good I can understand, from these examples, why someone would not be excited about them. How about a where clause that is not testing for equality? Does the conditions parameter to the “where” method still take an array? How about some examples of using “includes” to build more complex queries? It would be nice to see some queries in the Rails 2.x way and the Rails 3.x way in oder to compare readability. I do agree that the lazy loading will be a win for performance and/or allow better distinction between views and controllers.
Thanks for the nice writeup.
I want to caution against using table names directly, as in `cars.order(‘cars.price DESC’)`. In several different applications now I have worked with database administrators who want to refactor a database, and these queries make for brittle code. Much better is to do `cars.order(”#{Car.table_name}.price DESC”)`.
with_scope(where(:name => ‘lifo’) do
is missing a closing parenthesis. Awesome writeup Pratik, I’m really excited about using relational algebra. Speaking of which, the README on ARel looks like it is half-finished. It starts talking about “closure under joining even in the presence of aggregations” with an awesome example, but trails off? Who should I bug? :)
Awesome work dude! Can’t wait to write code against that API! :)
Looks brilliant, looking forward to use this. Thanks for the awesome work.
Do you have any real examples showing how this could be much more useful? It definitely seems nicer, but is a whole api revamp needed? What about some features to let us do “is not” , ”>” , ”<” or other things that would stop the use of query strings? WHat about adding some new features? The only new feature mentioned is the easier caching.
It just doesn’t seem a good enough improvement to depreciate all current rails apps.
@Chris sack up man, that should only take.. what.. a couple months to upgrade? What could be simpler?
You’ve got a dream, but I may not get to 3.1 with you.
@Pratik: Nice work, have been following these commits on github
Question: Does any of this address the object identity issues that DM and Sequel both handle? I’m speaking of the issue where @game.game_players.each{|gp| gp.game } will cause a re-fetch of the game object in AR 2.x, vs having a proper reverse association which enables child objects to refer to their parent.
Looks great, thank you for all the hard work!
Is there anything in the works to support combining conditions with OR in a single query? In the following scenario:
the call Item.red.white.all translates to “all items that are red AND white”, which would be an empty set. Is there a way to get all the Items whose color is red OR white in a single database query without using Item.find_by_sql(“SELECT * FROM items WHERE color = ‘red’ OR color = ‘white’”)?
Pratik, this would be a fresh addition! https://rails.lighthouseapp.com/projects/8994/tickets/2591-passing-in-fixnum-to-firstlast-finders#ticket-2591-14
@Nate At the moment Arel doesn’t help with what you want. The good news is that the :inverse_of options for associations that got added to ActiveRecord will (for simple cases like the one you mentioned).
Thanks, great work. I hope this will cut down on my current AR deadends
Pratik – This is really wonderful work. Favoring late binding proxies like this should help a lot of developers out with scaling issues.
One question about
Model#scoped, though. I’m not sure I see how it’s useful. ie: What doescars = Car#scopedgive you thatcars = Carwouldn’t already do? You can chain other scopes or#all,#first, etc. onto either of them.This is very helpful! Thank you for keeping the community informed.
These simple examples look fine, but can you give an example of how this would work when: * multiple tables are joined with conditions * more realistic WHERE clauses, e.g., (foos.x = ‘bar’ OR foos.x LIKE ‘bat)AND blobs.zip >= 0 * built-in db functions are needed * pagination of complex queries is used * subselects are required * one relies on Mysql fulltext search or Postgresql tsearch vectors or other data types (e.g., postgis coordinates)
In particular, paginating records grouped by by count of a relation was always a pain in the ass with Datamapper. At least with AR I can fall back into SQL joins and conditions to get the job done.
Pratik,
These new set of API’s are quite intuitive compared to the earlier set. They are sql’ish and hence we will know exactly what SQL query it will output the moment we code it. This is good.
What about default_scope?
class Foo < AR::Base default_scope order(“started_at DESC”).includes(:something) end
NoMethodError: undefined method `assert_valid_keys’ for #<activerecord::relation:0x102d5e578>
So, the big question is – what’s the performance like with this new API? Have any numbers?
@Alex : Thanks for pointing it out. It should work now and I’ve also updated the post.
@Tammer : We need scoped because a Relation caches the records. For example,
Without the concept of scoped, we’d need a new concept of a cache block.
@Vladimir : We did talk about the OR options, but decided not to introduce any new option for that for the 3.0 release. However, it should be simple to add an OR(|) operation on Relation, which would do exactly what you want. But it does get a little tricky when your relations have order, limit etc.
@Vladimir – I’m new to this new API, but I’d probably guess/go with something like this:
class Item scope :colored, lambda { |*args| where(:color => args)} end
Item.colored(“red”, “white”).all
It avoids using find_by_sql and is still readable.
With regards to OR queries, this is one instance where I wish ruby was a little more lispish, sure we could chain something like Item.where(:color => “red”).or.where(:color => “white”) together it’s not quite as nice as:
(where item (or (color :red) (color :white)))
I’m sure you could also do some magic with closures to clean it up a little more, but really OR queries using chained methods in ruby is going to be messy either way you look at it.
@Stefan – I actually much prefer your made up example of an ActiveRecord or syntax compared to Lisp’s equivalent. The lisp version is all out of order, it makes it sound like you’re coding in Yoda…
Looks.great(’,’).love().the()chainabilty(’!’)
THANK YOU (You know who you are!)
Being about to chain stuff will make ORs easier. It should make using the different condition styles (e.g. Array, Hash, String), a lot easier to maintain without having to use merge_conditions all over the place.
items = Item.where([“color = ? OR color = ?”, ‘red’, ‘white’]) items.where(:active => true AND :category_id => 37)
However, maybe it’d be simple to allow something like:
Item.where(:or, :color => ‘red’, :color => ‘white’)
Are there any plans for these methods to accept a block? (I realize you probably don’t want to duplicate Sequel functionality too much).
Don’t you think it’s a little ridiculous how Rails made many of us change all our “find_all”s to “find(:all)”, and now you’re making us change the “find(:all)”s back, to “all”?
This sort of capricious flip flopping might be really nifty refactoring funtime for rails’ hardcore code geeks, but it’s infuriating for this casual rails developer just trying to keep a couple of side projects current.
By the way this looks great and fantastic and magical overall; my comment is more directed at the overall “let’s break everything constantly” spirit I see in rails overall than at Pratik.
@RyanT : I wouldn’t really put this change in same category as find_all -> find(:all) -> all. Those were merely syntactical changes. But this is a whole new ballpark as it changes how finders work fundamentally and it won’t be fair to put it in the same category. With new finders, you will be calling all very rarely and likely to be relying on the lazy loading a lot more.
While I can understand your frustration, I’m pretty sure you’ll grow fond of the new way after working with it for a while, as it’s not only succinct but also a lot more flexible and poweful. And in any case, we’ll make sure the old way is supported long after 3.1 as well via an official plugin.
red_items = Item.where(:colour => ‘red’) red_items.find(1)
The way I understand it, it would return nil if Item.find(1) isn’t red. Is that what you meant? Shouldn’t it be red_items.first ?
Anyway, this looks great. I do hope this implementation will stay and we won’t have to recode all our apps for Rails 4! :D
Keep up the good work,
Eric
PS: I couldn’t find how to format code in comments. Any hint?
Awesome! You’d better call it “crazy loading” ;)
it’s time to deprecate old school apis, new api looks more clean & consistent, great work :)
These changes look like big improvements to active record, though I wonder if we shouldn’t just go all the way to creating a more expressive DSL that cuts out all of the inline SQL while we’re at it. Not to just pimp my own gem, but record_filter (http://github.com/aub/record_filter) creates a wrapper around the existing AR API that is expressive, readable, and cuts out all of the SQL string programming. I’d love to see more of its ideas integrated into active record.
This is awesome. Any idea of an official release date for rails 3?
pratik, can you explain which one API is correct for using new Advanced Query system in ActiveRecord
this:
Location.where(Arel::Attribute.new(Location.unscoped, :name).eq(“Warsaw”))
or this:
Location.where(Location.arel_table[:name].eq(“Warsaw”))
I understand that this is still pre code, and everything inside AR internals can still evolve, so it’s not recomended to use this API inside plugins code, but i just can’t wait to see this in action :)
Hi Pawel,
Neither of them is an official API. So you’re free to use either :)
@grosser and @seth I think what you’re taling about is something like
@people = People.where(:name=>‘Dave’) #No SQL called @people = @people.tall if @only_want_tall_people #Still no SQL?.... @people = @people.order(:name).limit(3)
then when you use @people as an enumerable either in controller or layout the SQL is fired so actually you can build a complex
so when I render a partial then the SQL is fied encompassing everything built on that Relation object? Is this right anyone?
I thought so, but are there any plans to add that API ? :) or Arel stuff will stay in deepest levels of ActiveRecord and be only available for the bravest plug-in coders :) ???
Hey Paweł,
There are no plans as such. But yeah, I suppose it’ll be more useful for the plugin developers than the actual application itself.
when my mom went away my father said fat birds cant fly
This is fantastic Pratik – definitely an improvement for a number of reasons – nice work.
One question – (I’ve been wanting to do this for ages with the old API but never got round to it – hopefully the new internals makes this easier):
Would it be easy to do something like this? :
class BlogPost scope :published, where(:published => true) end
Then, e.g. in a before_filter in my application:
BlogPost.make_scope_persistent(:published)
Then the ‘published’ scope would be there implicitly when you call other scopes etc.
I’ve seen people try to solve this kind of thing with default_scope but I’m convinced that’s the wrong place to do it, because you don’t necessarily always want to do it (e.g. in admin interface, in scripts, etc.), but you might want it everywhere in a specific part of your app (e.g. user-facing part of your site).
Thanks Mark
Mark, you could just do :
But please note that it’s not a recommended way. Ideally, you should just scoped all the find calls withBlogPost.published
Man, AR just became interesting again. I like how the interface is so much like Sequel.
I like this API. Also I was wondering, is it now possible to do something like: Post.select(‘users.email’, ‘posts.name’, ‘posts.description’).includes(:user).limit(10). So basically you can specify “exactly” what attributes from what tables should be fetched while using the .includes method, instead of always fetching “all” attributes? I believe this was not possible in AR2 but I might be wrong. Anyway, is it possible now?
Michael, that’s still not allowed and for good reasons. If you don’t want all the columns of the joined associations, you should just use :joins and not :include.
So it’s Sequel, but with a more recognizable name. I like it!
Pratik, concerning the last post of @Michael,
:joins implicitely uses INNER JOIN, while :include use LEFT OUTER JOIN. So while sometime we don’t want to :include other records we just want to LEFT OUTER JOIN.
So maybe will there be a way to choose the type of join ?
Another point: the SQL fragment in the order clause just smells bad, is there a plan to convert that to a more DSLish notation. Particullary when there’s table name aliasing in joins or include. Maybe something like order(:lastname => :desc) or with an included model includes(:items).order(:items => {:price => :asc})
Lastly, like some others already pointed it, will there be a documented way to hook in for plugin writers?
Thanx for the good job indeed.
Pascal.
@Pascal
1) I’m unsure about adding left_outer_join option to the finder methods. However, we should definitely make it easier for people to change the join type. As a temporary workaround, you could just use joins with strings.
2) I think we should wait before making DSL for every sql thing. Strings work just fine in many cases. I’d hate to end up with a DSL like sequel, which is very unfriendly and rather ugly for someone looking at it for the first time.
3) Yeah, plugin API is planned. But need to finalize the internals first. It should happen before 3.0 final.
Thanks!
This change will greatly improve code quality, reusability and readability imho. To be honest, one of the few things I never really understood about Rails was the lack of proper, chainable query builder methods on AR. Hash options felt somewhat awkward all along.
Is it awesome? Without some complex query examples, some documented notes of demonstrable advantages, and some hard and fast performance figures then might it just be pretty?
I have the feeling that there are too many deprecated things here, especially for those who got used to the find :all and find :first syntax. There is nothing wrong with this syntax, it works fine and is close to the generated SQL, which is not bad at all. And it is confusing to talk about deprecated things in 3.1 if 3.0 is not officially out yet.
awesome, I saw the DataMapper’s shadow
@Pratik,
Appears your missing ‘offset’ in the list of new finder methods.
@James : Added that. Thanks for the heads up.
Hi,
I wonder if it’s easier to add example Concrete Table Inheritance and such things in the rewrite (via Arel) or is’t the same like it was in 2.x?
Seems that I joined the party late :
). Looks really neat, delegations et al. Great going lifo… And, thanks a ton :)This is a great update. Although I have an enhancement suggestion.
While playing around with Arel I was surprised to find that there were no setters to modify a relation. In other words, “bang” methods (!) to update a relation using multiple calls. I think it would be nice if the following were possible:
users = Table[:users]
users.where!(users[:id].eq(1))
This would make it possible to use conditionals when generating a query.
I realize I can achieve a similar effect today just by storing the results of a where call into a new variable, but I think the bang methods are more elegant.
Thanks for your consideration.
Echoing everyone else, thanks so much for this work.
Can Arel math be applied to collections of objects, or only to records? That is, in:
articles = Article.by_user(user)
articles.published.each { |article| ... }
articles.unpublished.each { |article| ... }
is there any variation of this code that issues just one query without defining a ‘published?’ method on Article and partitioning on it? The example is trivial, but you can imagine that in more complex situations one would have to write a meaningful amount of logic once for Arel and once for working with objects in hand.
Nice to see a decent ORM for Rails at last – Django’s been having all these “new” goodies for decades. :)
Very nice, Pratik, and very useful. I’ve just been rewriting my queries in a new branch and all is well. Except for one thing.
While passing ( :id => id ) works just fine, how do we now handle inequality? i.e. doing [ ‘id != ?’, id ] as we can’t do ( :id != id ), can we? :D
It seems as though I can’t mix query syntax in a Model.where declaration, so how do I use the inequality operator in the new query?
Thanks again. Lovely work.
Ah, talking out loud often solves the problem. I simply throw in a :conditions hook like so:
Picture.order( ‘RAND’ ).where( :is_live => ‘1’, :story_id => self.id, :conditions => [ ‘id != ?’, current ] ).limit( 1 ).first
Actually you could just chain .where :
Removed limit(1) as .first would do it anyways
About “is not”, I suppose should be another method “where_not” for example, because query construction by hand is very painful and error-prone (and what if conditions are about join table?... code become a mess). I’ve looked at Arel attribute interface and mentioned only “eq” and “in” methods, there are no “not_eq”, “not_in” or something.
This is great, Pratik. Could this new ActiveRecord/ActiveSupport pair be released independently of Rails 3.0 for use with 2.x apps? Maybe make a Rails 2.4 release that is 2.3.5 plus new ActiveRecord/ActiveSupport?
That way existing apps could migrate in steps, if they’re not ready to upgrade all the way to 3.0 all at once (and the new ActiveRecord query interface is the most compelling reason to upgrade for most people, I think).
any thoughts on the comparison?
I wish searchlogic plugin survives this radical change in rails.
I am trying to migrate my rails application from 2.3.4 to rails 3.0.0 b. I am getting the following error while starting rails server. I google’ed it many times and not getting enough answers , struck up in this issue for 2 days , can u please help me in resolving this.
ravikud@ravikumard:~/src/mtnops$ rails server
AR==Datamapper ??
This seems very, very nice. A lot of sugar in my code now! =D But, what happens to find_by_* and find_all_by_* now? In Rails 3.0.0.beta, they work only with hash options. Do you know if they are going to be updated with where and its friends? Congrats for the amazing work!
Nice work, this is really inspiring, as I’m working on the ActiveRecord-like ORM for Java. Just a small correction:
The code:white_cars = cars.where(:colour => 'red')should probably look like this:red_cars = cars.where(:colour => 'red')Thanks much for all the hard work!
cheers
Awesome stuff. By vectron!
Does this mean that we can use is not null as a where condition?
For example: Cars.where(:colour => ‘red’).where(:num_wheels => “is not null”)
If not, is there going to be a simpler way to adding an ‘is not null’ query to the conditions for a model?
Is there a exhaustive documentation somewhere? I need more examples.
I thrw together some examples of using scope with conditions You can check them out at http://pastie.org/1132672
Thanks nice work. Very useful information to use as reference when migrating existing applications. Looking forward to use the new syntax.
Well worth the read. Thanks for sharing this information. I got a chance to know about this.
Great summary, just what i needed! Thanks!
Good job! Looking forward to use the new syntax. Thanks nice work. Very useful information to use as reference when migrating existing applications.
Thanks for this post. I’m using your post for AR and the rails_upgrade plugin. Thanks!!!
hi. I made a chinese edition of your article. http://thoughtrails.com/episodes/35-active-record-query-interface-for-rails3
The post is actually the freshest on this laudable subject. I harmonize with your conclusions and will thirstily look forward to see your approaching updates.
The router has been completely re-written and borrows heavily from the Merb router, but has a more developed DSL wga premie
I’ve been looking for the reviews on Rails 3, to get a better idea on the changes made. Thanks for your great effort to put things together!
Rails 3 will be great I hope that more people will start using it. webdesign terneuzen