Rushing to Judgement

I will begin this post with a disclaimer: my testing methods, so far, should be considered neither extensive nor scientific.

So, v1.0 of Transfer has been released. Transfer has been in development for several years, but Reactor was first on the scene as far as widely-used ORM frameworks for ColdFusion go. However, Alagad seems to be taking up all of Doug Hughes' time (Doug is the initial author of Reactor), and development past v1.0 release candidate 2 seems to have stagnated. Having said that, Reactor is still a solid product.

As lead developer for a system that uses Reactor, I have been somewhat disappointed to see the dropoff in interest in this framework. At the same time I do have a responsibility to investigate what other alternatives are available, and I've used Transfer for some small personal projects, although not in some time. I've also never done a side-by-side performance comparison between Reactor and Transfer. Sean Corfield did a nice job highlighting the basic differences between the two in a blog post last year.

Still, while Sean's post was interesting it did not talk about performance differences between the two frameworks. Indeed, prior to the release of Transfer, performance comparisons may have been premature. Now that Transfer is final - and the fact that the Reactor repository has been stable for a very long time speaks to it being "final" even if not officially so - we can probably make comparisons in a relatively fair way.

I have performed some simple tests to compare the performance of these two frameworks. The table used in the examples contains approximately 23k records. The database used is Microsoft SQL Server 2005 and the code was executed on ColdFusion 8.01 developer edition. I don't think getting into hardware details is warranted for this kind of test so I won't go over it. I used getTickCount() liberally to keep track of how long things were taking to run, and I ran each test at least a dozen times to calculate an average.

Important Configuration Details:

- I configured Reactor in production mode to conduct the tests.

- I configured Transfer with only the one table used in the tests (the database has a lot more than that).

- I did not use any of Transfer's caching mechanisms which most certainly would make a big difference for data that is frequently used in a real application.

Test 1. Compare performance for startup.

As most applications will be written to execute the ORM initialization routines only once, at application startup, this information may not be relevant; however, since I was recording it anyway, I decided to present it here, although it doesn't weigh into my subsequent analysis, and I don't think it should be a deciding factor in a framework choice.

Reactor: ~100ms (production mode) ~900ms (development mode)

Transfer: ~800ms

Test 1. Compare performance for Gateway/List operation.

This test involves creating a result set from all the rows in the table, without any select criteria. In Reactor we can do this with a Gateway object, while in Transfer the list() method serves the same purpose.

Reactor Average: ~600ms

Transfer Average: ~800ms

Test 2. Compare performance for load/read operation.

This test involves creating a business object populated from data already in the database. In Reactor a createRecord() followed by a load() was used. For Transfer the transfer.get() method was used. In each case loading was achieved using the primary key, which is an integer identity field in this particular table.

Reactor Average: ~130ms

Transfer Average: ~900ms

Test 3. Compare performance for update operation.

This test involves making a simple change to the data loaded in the second test and updating it back to the database. In Reactor we can do this with the save() method of the record. With Transfer you use transfer.save() and pass in the business object to accomplish the same thing.

Reactor Average: ~600ms

Transfer Average: ~1200ms

Conclusions and Next Steps

I was quite shocked to see how handily Reactor beat the pants off of Transfer in the performance department. Admittedly for things like the list() operation, caching is highly likely and would make a major difference; however, in my experience, most complex web applications are very CRUD-centric at the business object level, and if you expect a lot of unique transactions to take place in your application, you should not expect to see a big performance benefit from enabling caching at the business object level. Of course I welcome any differing opinion on this.

At a minimum, my initial observations warrant additional exploration. I'd like to do some more complicated tests involving relationships, and use load testing tools to observe other performance charactertics that could differentiate the two frameworks, and will do so in the next several weeks.