Combining all FEniCS projects into one shared repository

Registered by Martin Sandve Alnæs on 2013-02-06

We talked about combining all FEniCS projects into one shared repository. What are the pros and cons? It's a rather big operation, so it's best to be sure. Here's my take, please add other considerations.
I've copied a script below on the whiteboad which builds this shared repository from current trunks,
and the result can be browsed right now on lp:~martinal/+junk/unified-fenics-repo

Pros:
- Easy to fetch entire fenics sourecode
- Atomic merges of crosscutting changes
- Atomic updates on buildbot, avoiding the race condition to happens today if you
- Cleaner versioning
- Less release work
- Less confusion among users updating only some of the repos
- Blueprints, bugs, and discussions are often crosscutting and not associated with one single project
- Shared history of all projects (yes, I put that in pros)
- We can still keep the separated software architecture of the current projects
- But we can also choose to move all python modules into one site-packages/
- We can still keep the building and testing separated (but also add "buildall", "testall" eventually)

Cons:
- It's a big download just to get a single project like ufl or instant
- Slower bzr operations when working with just a small project like ufl
- Outstanding merges will be hard(er) and should be done before the repository merge, requiring some coordination
- .bzrignore files must be merged manually (not really a con, just a small boring job)
- The merged list of old tags is a bit messy (for future releases we can just use fenics-3.0 instead of 3.0)
- Backporting fixes will be tricky (worst case manual patching with diffs)
- For some projects it's unclear whether they should be part of FEniCS or standalone

As you see I land pretty strongly on the pro side now. We just need to avoid rushing things so we all agree there are no foreseeable problems, and execute this operation at a time where merges won't be a biggie.

To test how this will work out locally, get this branch or copy the script below and run it in a clean directory. Then watch the shared fenics/ repository be created from project trunks. To see the old history, try e.g.
  bzr log -rufc-history -n0 -l7
to see the last few commits of the ufc history. The history of single files is preserved.

I encourage somebody(tm) to try to figure out how to best handle merging of eventual outstanding feature branches into this new repository structure. Worst case is manual patching of files. I, for one, intend to avoid that issue.

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
None
Direction:
Needs approval
Assignee:
None
Definition:
Discussion
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

#!/bin/bash

# Select which projects to fetch here!
#PROJECTS='instant ufc ufl fiat ferari ffc dorsal uflacs'
PROJECTS='instant ufc ufl fiat ferari ffc dorsal uflacs dolfin'

# Create initial repository
mkdir fenics
pushd fenics
bzr init .
echo Welcome to the new and shiny shared repository for all FEniCS projects. > README
bzr add README
bzr commit -m'Welcome to the new and shiny shared repository for all FEniCS projects.'
popd

# For each project, branch it, move contents to subdir, and merge it into shared repo
mkdir projects
for p in ${PROJECTS}; do
    # Get fresh branch
    pushd projects
    bzr branch lp:$p

    # Move to subdir
    pushd $p
    ls -1 > filelist.tmp
    sed -i s/filelist.tmp// filelist.tmp
    mkdir $p-project
    echo bzr add $p-project
    bzr add $p-project
    echo bzr mv `cat filelist.tmp` $p-project/
    bzr mv `cat filelist.tmp` $p-project/

    if [[ -f .bzrignore ]]; then
        echo bzr mv .bzrignore $p-bzrignore
        bzr mv .bzrignore $p-bzrignore
    fi

    echo bzr commit -m"Move everything to directory "$p"-project"
    bzr commit -m"Move everything to directory "$p"-project"
    popd

    # Out of projects/
    popd

    # Merge into fenics
    pushd fenics
    echo bzr merge -r0..-1 ../projects/$p
    bzr merge -r0..-1 ../projects/$p
    echo bzr commit -m"Initial merge of "$p" into shared repository."
    bzr commit -m"Initial merge of "$p" into shared repository."
    echo bzr tag $p-history
    bzr tag $p-history
    popd
done

# Rename foo-project into just foo
pushd fenics
for p in ${PROJECTS}; do
    echo bzr mv $p-project $p
    bzr mv $p-project $p
    echo bzr commit -m"Rename "$p"-project to "$p"."
    bzr commit -m"Rename "$p"-project to "$p"."
done

# Show off the result
echo
echo Congratulations on your new and shiny FEniCS repository!
echo
bzr log
echo
echo Now try e.g.
echo bzr log -rufc-history -n0 -l10
echo to see the last 10 commits of the ufc history before The Merge.
echo
echo One last thing needs to be done: the old .bzrignore files are named
echo foo-bzrignore and must be manually rewritten into a single .bzrignore file.
echo

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.