diff options
| -rw-r--r-- | static/media/dev/merging-structural-changes/ideal-merge-results.png | bin | 0 -> 87122 bytes | |||
| -rw-r--r-- | static/media/dev/merging-structural-changes/mercurial-merge-results.png | bin | 0 -> 99313 bytes | |||
| -rw-r--r-- | static/media/dev/merging-structural-changes/subversion-merge-results.png | bin | 0 -> 114985 bytes | |||
| -rw-r--r-- | wiki/dev/liquibase.md | 77 | ||||
| -rw-r--r-- | wiki/dev/merging-structural-changes.md | 85 | ||||
| -rw-r--r-- | wiki/dev/twigs.md | 24 |
6 files changed, 186 insertions, 0 deletions
diff --git a/static/media/dev/merging-structural-changes/ideal-merge-results.png b/static/media/dev/merging-structural-changes/ideal-merge-results.png Binary files differnew file mode 100644 index 0000000..cb78110 --- /dev/null +++ b/static/media/dev/merging-structural-changes/ideal-merge-results.png diff --git a/static/media/dev/merging-structural-changes/mercurial-merge-results.png b/static/media/dev/merging-structural-changes/mercurial-merge-results.png Binary files differnew file mode 100644 index 0000000..d20f360 --- /dev/null +++ b/static/media/dev/merging-structural-changes/mercurial-merge-results.png diff --git a/static/media/dev/merging-structural-changes/subversion-merge-results.png b/static/media/dev/merging-structural-changes/subversion-merge-results.png Binary files differnew file mode 100644 index 0000000..5ef2570 --- /dev/null +++ b/static/media/dev/merging-structural-changes/subversion-merge-results.png diff --git a/wiki/dev/liquibase.md b/wiki/dev/liquibase.md new file mode 100644 index 0000000..6e5e97d --- /dev/null +++ b/wiki/dev/liquibase.md @@ -0,0 +1,77 @@ +# Liquibase + +Note to self: I think this (a) needs an outline and (b) wants to become a "how +to automate db upgrades for dummies" page. Also, this is really old (~2008) +and many things have changed: database migration tools are more +widely-available and mature now. On the other hand, I still see a lot of +questions on IRC that are based on not even knowing these tools exist. + +----- + +Successful software projects are characterized by extensive automation and +supporting tools. For source code, we have version control tools that support +tracking and reviewing changes, marking particular states for release, and +automating builds. For databases, the situation is rather less advanced in a +lot of places: outside of Rails, which has some rather nice +[migration](http://wiki.rubyonrails.org/rails/pages/understandingmigrations) +support, and [evolutions](http://code.google.com/p/django-evolution/) or +[South](http://south.aeracode.org) for Django, there are few tools that +actually track changes to the database or to the model in a reproducible way. + +While I was exploring the problem by writing some scripts for my own projects, +I came to a few conclusions. You need to keep a receipt for the changes a +database has been exposed to in the database itself so that the database can +be reproduced later. You only need scripts to go forward from older versions +to newer versions. Finally, you need to view DDL statements as a degenerate +form of diff, between two database states, that's not combinable the way +textual diff is except by concatenation. + +Someone on IRC mentioned [Liquibase](http://www.liquibase.org/) and +[migrate4j](http://migrate4j.sourceforge.net/) to me. Since I was already in +the middle of writing a second version of my own scripts to handle the issues +I found writing the first version, I stopped and compared notes. + +Liquibase is essentially the tool I was trying to write, only with two years +of relatively talented developer time poured into it rather than six weeks. + +Liquibase operates off of a version table it maintains in the database itself, +which tracks what changes have been applied to the database, and off of a +configuration file listing all of the database changes. Applying new changes +to a database is straightforward: by default, it goes through the file and +applies all the changes that are in the file that are not already in the +database, in order. This ensures that incremental changes during development +are reproduced in exactly the same way during deployment, something lots of +model-to-database migration tools have a problem with. + +The developers designed the configuraton file around some of the ideas from +[Refactoring +Databases](http://www.amazon.com/Refactoring-Databases-Evolutionary-Addison-Wesley-Signature/dp/0321293533), +and provided an [extensive list of canned +changes](http://www.liquibase.org/manual/home#available_database_refactorings) +as primitives in the database change scripts. However, it's also possible to +insert raw SQL commands (either DDL, or DML queries like `SELECT`s and +`INSERT`s) at any point in the change sequence if some change to the database +can't be accomplished with its set of refactorings. For truly hairy databases, +you can use either a Java class implementing your change logic or a shell +script alongside the configuration file. + +The tools for applying database changes to databases are similarly flexible: +out of the box, liquibase can be embedded in a fairly wide range of Java +applications using servlet context listeners, a Spring adapter, or a Grails +adapter; it can also be run from an ant or maven build, or as a standalone +tool. + +My biggest complaint is that liquibase is heavily Java-centric; while the +developers are planning .Net support, it'd be nice to use it for Python apps +as well. Triggering liquibase upgrades from anything other than a Java program +involves either shelling out to the `java` command or creating a JVM and +writing native glue to control the upgrade process, which are both pretty +painful. I'm also less than impressed with the javadoc documentation; while +the manual is excellent, the javadocs are fairly incomplete, making it hard to +write customized integrations. + +The liquibase developers deserve a lot of credit for solving a hard problem +very cleanly. + +*[DDL]: Data Definition Language +*[DML]: Data Manipulation Language
\ No newline at end of file diff --git a/wiki/dev/merging-structural-changes.md b/wiki/dev/merging-structural-changes.md new file mode 100644 index 0000000..f597d39 --- /dev/null +++ b/wiki/dev/merging-structural-changes.md @@ -0,0 +1,85 @@ +# Merging Structural Changes + +In 2008, a project I was working on set out to reinvent their build process, +migrating from a mass of poorly-written Ant scripts to Maven and reorganizing +their source tree in the process. The development process was based on having +a branch per client, so there was a lot of ongoing development on the original +layout for clients that hadn't been migrated yet. We discovered that our +version control tool, [Subversion](http://subversion.tigris.org/), was unable +to merge the changes between client branches on the old structure and the +trunk on the new structure automatically. + +Curiousity piqued, I cooked up a script that reproduces the problem and +performs the merge from various directions to examine the results. Subversion, +sadly, performed dismally: none of the merge scenarios tested retained content +changes when merging structural changes to the same files. + +## The Preferred Outcome + + + +The diagram above shows a very simple source tree with one directory, `dir-a`, +containing one file with two lines in it. On one branch, the file is modified +to have a third line; on another branch, the directory is renamed to `dir-b`. +Then, both branches are merged, and the resulting tree contains both sets of +changes: the file has three lines, and the directory has a new name. + +This is the preferred outcome, as no changes are lost or require manual +merging. + +## Subversion + + + +There are two merge scenarios in this diagram, with almost the same outcome. +On the left, a working copy of the branch where the file's content changed is +checked out, then the changes from the branch where the structure changed are +merged in. On the right, a working copy of the branch where the structure +changed is checked out, then the changes from the branch where the content +changed are merged in. In both cases, the result of the merge has the new +directory name, and the original file contents. In one case, the merge +triggers a rather opaque warning about a "missing file"; in the other, the +merge silently ignores the content changes. + +This is a consequence of the way Subversion implements renames and copies. +When Subversion assembles a changeset for committing to the repository, it +comes up with a list of primitive operations that reproduce the change. There +is no primitive that says "this object was moved," only primitives which say +"this object was deleted" or "this object was added, as a copy of that +object." When you move a file in Subversion, those two operations are +scheduled. Later, when Subversion goes to merge content changes to the +original file, all it sees is that the file has been deleted; it's completely +unaware that there is a new name for the same file. + +This would be fairly easy to remedy by adding a "this object was moved to that +object" primitive to the changeset language, and [a bug report for just such a +feature](http://subversion.tigris.org/issues/show_bug.cgi?id=898) was filed in +2002. However, by that time Subversion's repository and changeset formats had +essentially frozen, as Subversion was approaching a 1.0 release and more +important bugs _without_ workarounds were a priority. + +There is some work going on in Subversion 1.6 to handle tree conflicts (the +kind of conflicts that come from this kind of structural change) more +sensibly, which will cause the two merges above to generate a Conflict result, +which is not as good as automatically merging it but far better than silently +ignoring changes. + +## Mercurial + + + +Interestingly, there are tools which get this merge scenario right: the +diagram above shows how [Mercurial](http://www.selenic.com/mercurial/) handles +the same two tests. Since its changeset language does include an "object +moved" primitive, it's able to take a content change for `dir-a/file` and +apply it to `dir-b/file` if appropriate. + +## Git + +Git also gets this scenario right, _usually_. Unlike Mercurial, Git does not +track file copies or renames in its commits at all, prefering to infer them by +content comparison every time it performs a move-aware operation, such as a +merge. diff --git a/wiki/dev/twigs.md b/wiki/dev/twigs.md new file mode 100644 index 0000000..ebc875c --- /dev/null +++ b/wiki/dev/twigs.md @@ -0,0 +1,24 @@ +# Branches and Twigs + +## Twigs + +* Relatively short-lived +* Share the commit policy of their parent branch +* Gain little value from global names +* Examples: most "topic branches" are twigs + +## Branches + +* Relatively long-lived +* Correspond to differences in commit policy +* Gain lots of value from global names +* Examples: git-flow 'master', 'develop', &c; hg 'stable' vs 'default'; + release branches + +## Commit policy + +* Decisions like "should every commit pass tests?" and "is rewriting or + deleting a commit acceptable?" are, collectively, the policy of a branch +* Can be very formal or even tool-enforced, or ad-hoc and fluid +* Shared understanding of commit policy helps get everyone's expectations + lined up, easing other SCM-mediated conversations |
