fallenrogue.com

Keep that SQL in Ruby or doing it right with migrations

I’ve been using rails for almost a year now (my how the time flies…) and in thinking of things I’d like to share from my findings with Rails to the community one thing came to mind. It’s something that comes up a lot on blogs and although I wasn’t sure if I wanted to repeat what has been said, I figured why the hell not? After all, I think it’s important: rails migrations. Almost everything I’m gonna go over is in the API docs here so if you prefer to DIY have at!


Migrations are those little ruby files that are added to the db/migrate/ folder every time you create a model. If you haven’t looked in there, go ahead and check it out, they are there. They don’t have much in them, but they are there just waiting to be used. There are many seasoned developers out there who can sling t-sql like champions, I count myself as one of them, for us, it’s very difficult to consider breaking the n-tier abstraction principals of OO (especially when we know what we’re doing) and simply “trust” Ruby and Rails. If you can give into your fears for a little bit, you’ll see that there are certainly great rewards to using migrations.


Ruby: the one to rule them all. I want you to take a leap of faith and know the following. If you’re using Ruby on Rails, you’ve committed to the Rails way and should follow its conventions, explicitly. Dealing with your database is no different. After all, your database is a part of your Rails application, isn’t it? Just because you had to manually trudge through a mountain of t-sql before Rails doesn’t mean you still need to. Unless you’re addicted to the DB GUI crack… then you’re on your own to find rehab. I can’t help you. You trust Rails to make your calls to the database, to validate the data before it goes it, to deliver it back to you quickly when it comes out, right? Well, why not trust it to create your schema as well? This way if you need to deploy your entire application from Unix/MySQL to Windows/MSSQL you can and you won’t even break a sweat. You’re not locking yourself into DB specific syntax. You’re using Ruby to talk to any Rails supported DB. It’s what the big boys call being “database agnostic”.


Enough Jibber-Jabber! Let’s sling that code! You’ve read this far, which means you want to see some migrations in action, so let’s get to it! This time, I’m placing the code RIGHT HERE in the middle of the article so you can follow along! Don’t worry, it’s at the end if you want to type(Copy and Paste) your way through the examples. Let’s start at the beginning…with a new project. This project is going to be the “start” of what would be (what might be…) a bigger project. The focus in on migrations and not what the app will actually do. :) Be sure to create your databases as well. I’m using Sqlite3… so that step is done for me

 
rails miapp 
(if using sqlite3) rails miapp -d sqlite3

Now, let’s create a few models that we are going to use in our app. I’m going to create models that I have created for every app I’ve ever written (I’ve got a generator for them now, but that’s a different article.) so they may look familiar to you
ruby script/generate model user
ruby script/generate model role

you can probably guess where this is going. But instead of one role for each user, lets edit our models to allow a has_and_belongs_to_many relationship between the models. This means that my user can have many roles.
class User < ActiveRecord::Base
    has_and_belongs_to_many :roles
end

class Role < ActiveRecord::Base
    has_and_belongs_to_many :users
end

So far, nothing out of the ordinary. At this point, some of you would go to your mysql, sqlite3, etc command line and start typing out a sql statement to create the 3 tables needed to make this happen. Or, you’re like many developers you launch a GUI app and create the tables there, adding new relationships and keys… letting the GUI build the SQL for you. For you, this should be a smaller leap of faith than the people constructing them by hand. If you will trust a GUI with your sql statements why not rails? After all, you trust them with all the other sql queries…what’s a few more?

So, we know that 2 tables have already been created but if we’re going to allow multiples roles per user we are going to need another table to relate the users and roles tables. To get this migration we type the following into the terminal (please note the habtm alphabetical convention)
ruby script/generate migration roles_users

Ok, the meat and potatoes. let’s open those migrations and add some simple properties to the migration files. Open the first file which if you’re following along would be db/migrate/001_create_users.rb
class CreateUsers < ActiveRecord::Migration
  def self.up
    create_table :users do |t|
      # t.column :name, :string
    end
  end

  def self.down
    drop_table :users
  end
end

Look familiar? It should, it’s RUBY! so, without changing languages, we’re going to set up our database tables following the habtm(has_and_belongs_to_many) convention. Here are the migration files as I have created them,
class CreateUsers < ActiveRecord::Migration
  def self.up
    create_table :users do |t|
      t.column :name, :string
    t.column :password, :string
    t.column :email, :string
    end

    User.create( :name=>"admin", :password=>"admin",  email=>"admin@admin.com")
  end

  def self.down
    drop_table :users
  end
end

 class CreateRoles < ActiveRecord::Migration
  def self.up
    create_table :roles do |t|
      t.column :name, :string
    end

    %w{admin user}.map{|role| Role.create(:name=>role)}
  end

  def self.down
    drop_table :roles
  end
end

class RolesUsers < ActiveRecord::Migration
  def self.up
    create_table :roles_users, :id=> false do |t|
        t.column :role_id, :integer
        t.column :user_id, :integer
    end

    User.find_by_name("admin").roles << Role.find_by_name("admin")
  end

  def self.down
        drop_table :roles_users
  end
end

OK. There’s a lot in there but the things I want you to see right off the bat, is that these are Ruby classes. This is the same language that you’ve been writing your model and controllers in. Migration run as the application so you have full access the models that you’ve created.


If you notice, after each of the create_table methods is called I have established some default values using Ruby. That way, when I start up my application I’ll have to data to play with. When all else fails, you can also insert a Role by opening the script/console and doing it there. Could I have done this in t-sql? Of course, but why do you need to? I haven’t created things like indexes and key relationships because Rails keeps that logic in the business layer instead of the data layer. It is certainly a hot topic for debate (esp amongst DBAs!) but that debate is better served in a different forum.


So, now, all you have to do to get this running is open that terminal window and run…
rake db:migrate

You’re done. You’ve just created a schema for your models and even ran code that proves that they all work! If you don’t believe me you can scaffold a new controller set for users and roles and CRUD away! Or you can go to the console and look for the user and roles. Or Add and remove roles from the user. It’s alot of fun. I did this in sqlite3. But now if I wanted to run it in mysql, I would just reconfigure my database.yml file to point to mysql and run rake db:migrate and BAM, I’m ready to go!


I haven’t gone over nearly all of the options when it comes to migrations but between the sample code and the API you can see all of the additional things that can be done to keep your feature complete when creating your database schema for your application.


Oh hell, just for fun let’s assume that we’re not done yet. Someone came by 10 months after we’ve released to Production and wants to add a few more columns to the user model. They are optional columns but wants users to fill them in if they want. They are blog url and hometown. So, instead of cracking open the database or running a query to modify the tables through sql we’re going to do a migration. Let’s go…
ruby script/generate migration add_blog_url_and_hometown_to_users

How do you like that long name? It’s crappy but you get the point. Let’s add those columns to the user table
class AddBlogUrlAndHometownToUsers < ActiveRecord::Migration
  def self.up
    add_column :users, :blog_url, :string
    add_column :users, :hometown, :integer
  end

  def self.down
    remote_column :users, :blog_url
    remove_column :usres, :hometown    
  end
end

then run the rake…
rake db:migrate

done. OH NO!!!! I fracked it up again! hometown is not supposed to be an integer! It’s supposed to be a string! I guess I’m going to have to go into the database after all, right? Wrong. Just roll back to the previous version make the change and migrate up again.
rake db:migrate VERSION=3

class AddBlogUrlAndHometownToUsers < ActiveRecord::Migration
  def self.up
    add_column :users, :blog_url, :string
    add_column :users, :hometown, :string
  end

  def self.down
    remote_column :users, :blog_url
    remove_column :usres, :hometown    
  end
end

rake db:migrate


So…how did I know to roll back to VERSION=3? see those numbers in front of the migration file names? Those are your versions. If you remove one, you may want to refactor them to keep them up to date, but that’s completely up to you.


So, that’s migrations in a nutshell. Migrations are a great way to use Ruby to create your database schema in a quick and easy fashion. I didn’t go over tests or deploying to production or any of those things (dumping, importing schema etc) those are for another day. For now, just know that migrations are a clean DRY way to manage your schema in Rails and if you’re not taking advantage of them you might want to take another look!


Download the project files here.

articleStats

Here are some silly little facts about this Keep that SQL in Ruby or doing it right with migrations...

It was written by about 1 year ago.
It has 11862 letters in it.
It has 1725 words in it.
It has a total of 11 comments in all.
So far fallenrogue has the last word!

article Links

These are the links that appear in this article. They probably don't make sense out of context... but just in case. :)

<span class="caps"><span class="caps"><span class="caps">API</span></span></span> docs here
<span class="caps"><span class="caps"><span class="caps">RIGHT HERE</span></span></span>
Download the project files here.
 

The other stuff...

What the kids are saying...

about 1 year ago K. Adam Christensen said...

One thing you may want to think about is instead of creating data in your migrations, like when you created new Roles and a User, use data that you have in your fixtures.

Since you're using migrations and you want to do things the rails way, then you should also be doing unit testing. Unit testing uses your test database, and that database will not run migrations. It will use your finished schema. This means that the data that you have just added by using the migrations will not transfer to your test database. So in order to use your unit tests with data, you need to have fixtures. These fixtures could contain test users, like your admin user, as well as the roles that you have assigned in your migrations.

To me it goes against all that is DRY to have data in your migrations and data in fixtures when it's the same thing. So I have written a blog entry on "how to load your fixtures from your migrations":http://shifteleven.com/articles/2006/09/28/loading-fixtures-in-a-migration

And as an aside, I know the purpose of going back a version with your migrations was to show how that functionality works; however, I think it's bad form to do so as a practice. Like I said, it works for this case, but if this code were committed to a repository and then other users ran that migration before you went in to make a change, then there could be unnecessary bug tracking going on. Instead, it's better to create a new migration which fixes something that was in the previous migration(s).

about 1 year ago fallenrogue said...

@K.Adam - Ah yes, FANTASTIC comment! I debated and debated whether or not to go the fixtures route in this article. In fact, I actually had but decided to keep this a 100 level and do a follow up using Fixtures, but you seem to have beaten me too it! :) <br>

that said, there are some excellent points in your comment but I believe that it's good to know all of the available options to you as your hit the ground running with Rails. When experimenting as a beginner I found the revision rollbacks to be a wonderful learning tool and wanted to share them here. More to your point, if you were committing a very dangerous change to an established database then rolling back would be a terrible thing to do, no matter what, especially if there is production data on the line, but when you're playing around a bit, it's nice to know that it's there rather than going into your Sql editor of choice, dropping the table, modifying the schema table back a version and running rake migrate again all because you mistyped something. That's where I was coming from and perhaps could be a little more clear on. I'll make note of that for future articles. Thanks so much for stopping by and providing your thoughts as well as the resource on Fixtures!

about 1 year ago Jacob Atzen said...

I don't really like the idea of using test fixtures as a way to load data into your production application. As your test suite and fixtures grow you'll have more and more data in your fixtures which you will most likely not want to have in your application. As such I would advocate loading data for the application directly from the migrations. If you wan't to keep it in yaml or something else fine, just don't mix test data with live data.

about 1 year ago fallenrogue said...

Jacob, your reason is exactly why I didn't use Fixtures in my example. Often times I want to load a bunch of data from my YAML fixtures into Dev to play with it without typing it again. But this rarely the case in production. Like in my user example, I will always add an "admin" user to dev and production then delete them later. But in my test fixtures I have about 10 users to test for roles of each users, adding friends, removing friends, banning spammers, etc. I would never want to load that into my production db, ever.

about 1 year ago Jason said...

Two comments: a statement and a question:

Statement:

Since migrations are sequential, I don't think it can take into account:
1. database version 1 had a field called email.
2. database version 2 deleted this field
3. database version 3 reinstated this field

If you want to upgrade from version 1 to version 3 and that field had data in it for multiple records, the migrations sequential upgrade won't carry that data through. The email field will be blank in version 3.

Question:

If I want to change the column type in Oracle, the alter command will work. However, the alter command won't work if that field has data in it. An insert-select is needed (i.e. create temp table, insert data from old table into new temp table, delete old,original table, rename temp table name to that of old, original table). Will the migrations SQL abstraction handle this schema change on the column and preserve the record data therein?

about 1 year ago fallenrogue said...

Yes, if you remove a column and then add it back a couple of revisions later... that will destroy the data because you destroyed the column and all the data therein. So, be careful what you call and when. <br>

Your question about redefining the data type for the column, I can't answer for Oracle cause I've never used Rails with Oracle so I don't know if there are quirks, I can say that in mysql and sqlite3 if you call the change_column command and you change between two similar types the data remains. If you change from a string to an integer column (I haven't tested this) I assume the db would simply truncate the data because it's not of the correct data type. but if you were to say <br>

change_column :users, :email, :text<br>

then your :string field will become a :text field (Rails style) without losing the data. Did that answer the question? thanks!

about 1 year ago Jason said...

very interesting, thanks that helped!

about 1 year ago navinsamuel said...

Great article and comments. Does anyone know how to create database views using rake?

about 1 year ago fallenrogue said...

@navinsamuel: thanks for the feedback. Question: Why would you create views? I would say if you need views for other users of your database (and not your applications) then that may be the time to go directly to your db to create them. But if you're staying in Rails and only giving people access to the data through your application there is so much that you can do to join and combine data from multiple tables into objects (or models) that you should never have to. (my next article is going to do some of that!)<br>

Rails is full stack which means all tiers of your app are managed through the Rails framework. If that is the case, then a view is largely unnecessary; much like stored procedures are unnecessary in Rails. Rails handles all of that and you can optimize your queries from rails as needed. <br>

I will say that there is one database convention that you may want to implement that does not make itself easily known in Rails and that's indexes. You can easily create and drop indexes in Rails to keep those look ups fast. So if I wanted to add an index to the users table on the email field (or is it email_address field at this time? I can't remember) Just do this...<br>

add_index(:users, :email)<br>

I hope I've answered your question, if not please feel free to provide an example and elaborate on your specific issue. Thanks!

about 1 year ago jt said...

i just wish migrations supported things like unsigned, bigints, and the likes :(

about 1 year ago fallenrogue said...

Maybe someday! ;)

Leave a comment
*name:
*email: (never sold or published.)
url :

©2000-2008 fallenrogue.com | Some Rights reserved.