Be able to build injectors on dedicated build machines

Registered by Peter Banka

(09:58:11) peter.banka@joseywales/3397bf6a: So how about that build server idea...?
(09:58:58) shawn-jabber: For building what?
(09:59:10) peter.banka@joseywales/3397bf6a: anything that needs compiling
(09:59:27) shawn-jabber: for centos stuff?
(09:59:46) peter.banka@joseywales/3397bf6a: or ubuntu, but that's probably less interesting to us atm
(10:00:07) peter.banka@joseywales/3397bf6a: It could be for any injector pre-processing
(10:00:13) peter.banka@joseywales/3397bf6a: or dists
(10:00:17) shawn-jabber: What in particular are you trying to do?
(10:00:57) peter.banka@joseywales/3397bf6a: Sometimes you want to be able to deploy code for which you only have a source tarball.
(10:01:26) shawn-jabber: and you want to avoid building it on the client
(10:01:29) peter.banka@joseywales/3397bf6a: yeah
(10:01:42) peter.banka@joseywales/3397bf6a: could be a solution for the pycrypto problem, as well
(10:01:44) shawn-jabber: I think we had a conversation about this the other day
(10:01:49) peter.banka@joseywales/3397bf6a: we did
(10:02:42) shawn-jabber: Okay, so are we going to tie it in somehow directly to bombardier or just use it to generate injectors?
(10:03:19) peter.banka@joseywales/3397bf6a: I was thinking of tie-ing it into... but it can be a future roadmap thing... not necessarily something to tackle right now
(10:03:48) shawn-jabber: okay. I'll get a centos one going then
(10:03:58) peter.banka@joseywales/3397bf6a: I think the overall appeal of Bombardier would be a lot better if Bombardier could compile.
(10:04:10) peter.banka@joseywales/3397bf6a: You could distribute generic open-source packages a lot easier that way
(10:07:13) shawn-jabber: There'd be some plumbing involved, but I think we could make that work. If you had some metadata like which machine would build for a platform. I'll keep that in mind for a while and see what happens ideawise
(10:08:36) peter.banka@joseywales/3397bf6a: Yeah, cool. I think that you could specify for each package injector you could specify one or more platforms for a package and associate a build script
(10:09:39) shawn-jabber: Maybe the build step for a package could take an address like root@buildbox:/var/build/blah.tar.gz for the injector path
(10:09:59) shawn-jabber: if it didn't already have it on the box or something
(10:10:21) peter.banka@joseywales/3397bf6a: hmm
(10:10:36) shawn-jabber: Like escalating metadata, from path to remote path to build command or something
(10:10:45) peter.banka@joseywales/3397bf6a: don't follow you...
(10:11:20) shawn-jabber: Kind of how a makefile works in that if you have what the target requires you use it. Otherwise you know how to get it built with other rules
(10:11:22) peter.banka@joseywales/3397bf6a: like this ?
class_name: PythonInstaller.PythonInstaller
configuration: {}
dependencies: []
executables: [configure, install, uninstall, verify]
injectors:
  Python-2.6.4:
    path: /var/deploy/repos/injectors/Python-2.6.4.tar.gz
    platforms: [centos5.4]
    build_script: root@buildbox:/var/build/blah.tar.gz
libs:
  AptPackageInstaller: {svn: 'https://svn.ge-is-beta.com/icontrol/src/trunk/libs/AptPackageInstaller',
    version: '850'}
  GenericInstaller: {svn: 'https://svn.ge-is-beta.com/icontrol/src/trunk/pkgs/GenericInstaller', version: '850'}
  PythonInstaller: {svn: 'https://svn.ge-is-beta.com/icontrol/src/trunk/pkgs/PythonInstaller', version: '850'}
  YumPackageInstaller: {svn: 'https://svn.ge-is-beta.com/icontrol/src/trunk/libs/YumPackageInstaller',
    version: '850'}
package-version: 5
priority: 500
release: 12
(10:12:35) shawn-jabber: instead of build script, that would be the remote path to it maybe
(10:13:08) peter.banka@joseywales/3397bf6a: I would think you'd want an SVN path to your build script
(10:13:45) shawn-jabber: I meant that the tarball would be a finished injector
(10:14:07) shawn-jabber: remote_path: root@buildbox:/var/build/blah.tar.gz being the injector tarball
(10:14:49) peter.banka@joseywales/3397bf6a: how about first defining your build servers...:
/var/deploy/build_servers

ubuntu-hardy.yml:
  ip_address: 10.3.4.5
  default_user: root
  build_directory: /var/build

centos5-4.yml:
  ip_address: 10.3.4.6
  default_user: root
  build_directory: /var/build
(10:15:08) shawn-jabber: Sure
(10:15:20) shawn-jabber: I think that's good
(10:17:14) peter.banka@joseywales/3397bf6a: then could you define the package this way:
class_name: PythonInstaller.PythonInstaller
configuration: {}
dependencies: []
executables: [configure, install, uninstall, verify]
injectors:
  Python-2.6.4:
    source: /var/deploy/repos/injectors/Python-2.6.4.tar.gz
    platforms: [centos5.4]
    paths:
       - /var/deploy/injector/python-2.6.4-centos52.tar.gz
    build_script: https://svn.ge-is-beta.com/var/build/blah.sh
libs:
  AptPackageInstaller: {svn: 'https://svn.ge-is-beta.com/icontrol/src/trunk/libs/AptPackageInstaller',
    version: '850'}
  GenericInstaller: {svn: 'https://svn.ge-is-beta.com/icontrol/src/trunk/pkgs/GenericInstaller', version: '850'}
  PythonInstaller: {svn: 'https://svn.ge-is-beta.com/icontrol/src/trunk/pkgs/PythonInstaller', version: '850'}
  YumPackageInstaller: {svn: 'https://svn.ge-is-beta.com/icontrol/src/trunk/libs/YumPackageInstaller',
    version: '850'}
package-version: 5
priority: 500
release: 12
(10:17:55) shawn-jabber: I think that works
(10:17:57) peter.banka@joseywales/3397bf6a: instead of a "path" for an injector, if you define "platforms" then you would have to have a "paths" section instead, one file per platform
(10:18:42) shawn-jabber: Maybe those could be tied together as a dictionary value under the platform like
(10:18:56) peter.banka@joseywales/3397bf6a: oh, right
(10:19:21) peter.banka@joseywales/3397bf6a:
injectors:
  Python-2.6.4:
    source: /var/deploy/repos/injectors/Python-2.6.4.tar.gz
    platforms:
      centos5.4: /var/deploy/injector/python-2.6.4-centos52.tar.gz
    build_script: https://svn.ge-is-beta.com/var/build/blah.sh

(10:19:54) shawn-jabber: yeah, that's cool. The build script could maybe be overridden per platform if it was necessary too
(10:20:11) peter.banka@joseywales/3397bf6a: right...
(10:20:26) shawn-jabber: That's best left to later when we see how we need it to be though
(10:20:37) peter.banka@joseywales/3397bf6a: so you what this then:
injectors:
  Python-2.6.4:
    source: /var/deploy/repos/injectors/Python-2.6.4.tar.gz
    platforms:
      centos5.4:
         path: /var/deploy/injector/python-2.6.4-centos52.tar.gz
         build_script: https://svn.ge-is-beta.com/var/build/blah.sh
(10:20:46) peter.banka@joseywales/3397bf6a: might as well
(10:21:24) shawn-jabber: Sure, if we don't like it, changing it should be easy
(10:21:55) peter.banka@joseywales/3397bf6a: right... worse case is you have to duplicate the build_script entry...
(10:22:29) shawn-jabber: Yeah, also we might need to version stamp the path to the end product
(10:22:51) shawn-jabber: That could be part of the build process though to return the path to the newly built thing
(10:23:00) peter.banka@joseywales/3397bf6a: right.
(10:23:27) shawn-jabber: I also think it should be lazy about building if it has a suitable one locally
(10:23:47) peter.banka@joseywales/3397bf6a: hmm..
(10:24:07) shawn-jabber: when you build the package, if you already have an injector that is okay then don't build
(10:24:11) peter.banka@joseywales/3397bf6a: right, if the path is there it should not build the injector every time you build the package
(10:24:28) shawn-jabber: yeah
(10:25:50) peter.banka@joseywales/3397bf6a: Do you want to go this way for all packages? Like this:
class_name: TouchscreenFlasher.TouchscreenFlasher
configuration:
  TouchscreenFlashStatus: {admin_email: <email address hidden>, daemon_group: www-data,
    daemon_user: www-data, log_dir: /var/log/apache2, server_port: '443'}
  ip_address: ''
  touchscreen_flasher: {debug: 'True', image_directory: /opt/touchscreen_flash, image_filename: IMX27ADS_nb_iplspl_0x000C0000.bin,
    touchscreen_password: '123456', touchscreen_user: root}
dependencies: [Django-1-1-1]
executables: [configure, install, restart, start, stop, uninstall, verify]
injectors:
  Flasher: {
    generic:
       path: '/var/deploy/repos/injectors/Flasher-851.tar.gz',
       source: 'https://svn.ge-is-beta.com/icontrol/src/trunk/injectors/Flasher'
       version: '851'
(10:26:33) peter.banka@joseywales/3397bf6a: Use the "generic" target if it doesn't need platform-specific building? And if there's no build script, it just graps the stuff from source?
(10:27:34) shawn-jabber: If we can keep the existing packages from need to all be rebuilt that would sure be nice
(10:28:42) peter.banka@joseywales/3397bf6a: yeah... re-arranging this way I don't think would require a re-build
(10:28:46) shawn-jabber: I like having a path at the top level being the first place it looks
(10:28:51) peter.banka@joseywales/3397bf6a: okay
(10:29:05) shawn-jabber: injectors:
  ic_server: {path: IConnect_v3.4.2.3-0-r1108.tar.gz}

(10:29:26) shawn-jabber: if add a build key at the same level as path, it could look there next
(10:29:36) peter.banka@joseywales/3397bf6a: don't follow you
(10:30:08) shawn-jabber: ic_server: {path: IConnect_v3.4.2.3-0-r1108.tar.gz, build: {bar, baz, etc}}
(10:30:40) peter.banka@joseywales/3397bf6a: build doesn't make sense without a platform context, I don't think
(10:30:54) shawn-jabber: That's baz, bar and whatever
(10:31:16) peter.banka@joseywales/3397bf6a: so like this:
class_name: PythonInstaller.PythonInstaller
configuration: {}
dependencies: []
executables: [configure, install, uninstall, verify]
injectors:
  Python-2.6.4:
    source: /var/deploy/repos/injectors/Python-2.6.4.tar.gz
    build:
      platforms:
        centos5.4:
           path: /var/deploy/injector/python-2.6.4-centos52-24.tar.gz
           build_script: https://svn.ge-is-beta.com/var/build/blah.sh
(10:31:33) peter.banka@joseywales/3397bf6a: or you want to substitute "platforms" with "build"
(10:31:58) peter.banka@joseywales/3397bf6a: injectors:
  Python-2.6.4:
    source: /var/deploy/repos/injectors/Python-2.6.4.tar.gz
    build:
      centos5.4:
         path: /var/deploy/injector/python-2.6.4-centos52-24.tar.gz
         build_script: https://svn.ge-is-beta.com/var/build/blah.sh

(10:32:16) shawn-jabber: I think build can have all that stuff under it just fine. I want a path key at the same level as source and buld
(10:32:17) shawn-jabber: build
(10:32:26) shawn-jabber: like it is now
(10:32:37) peter.banka@joseywales/3397bf6a: but if "build" is there, you won't have one
(10:32:51) peter.banka@joseywales/3397bf6a: "path" will have to be under "build"
(10:33:23) shawn-jabber: I think that path should be up there with build and source, and if it's not found there it should start looking at how it might build it
(10:33:46) peter.banka@joseywales/3397bf6a: okay... but after it's done building it still won't have a path at the top-level.
(10:34:03) shawn-jabber: I'd put it there if I wanted it
(10:34:23) peter.banka@joseywales/3397bf6a: fair enough. But I imagine a package could support multiple platforms
(10:34:42) shawn-jabber: I don't want to rebuild anything complicated if I just have a static injector tarball sitting in the repos
(10:34:53) peter.banka@joseywales/3397bf6a: sure
(10:35:01) shawn-jabber: I still like having a path to it as a default
(10:35:13) peter.banka@joseywales/3397bf6a: right
(10:37:10) shawn-jabber: I like being able to specify the build though. Really, if you had that default path and it wasn't there, when you did the build you could copy it to that path
(10:37:50) shawn-jabber: next time you'd just grab it
(10:38:09) peter.banka@joseywales/3397bf6a: okay, but how would it work in this case...:
class_name: PythonInstaller.PythonInstaller
configuration: {}
dependencies: []
executables: [configure, install, uninstall, verify]
injectors:
  Python-2.6.4:
    source: /var/deploy/repos/injectors/Python-2.6.4.tar.gz
    build:
      centos5.4:
         path: /var/deploy/injector/python-2.6.4-centos52-24.tar.gz
         build_script: https://svn.ge-is-beta.com/var/build/blah.sh
      ubuntu-hardy:
         path: /var/deploy/injector/python-2.6.4-ubuntuhardy-24.tar.gz
         build_script: https://svn.ge-is-beta.com/var/build/blah.sh

(10:38:28) peter.banka@joseywales/3397bf6a: doesn't keep you from "just grabbing it", but you need to specify a path per platform
(10:38:59) shawn-jabber: If you make it for what you're doing and don't break my top level path in the ones I'm using it's fine with me
(10:39:19) peter.banka@joseywales/3397bf6a: yeah, we already agreed to that
(10:39:27) shawn-jabber: I think it's good like that then
(10:40:37) shawn-jabber: With these really simple third party ones having a definite path is fine. Anything custom built will require us to allow for custom names.
(10:41:00) peter.banka@joseywales/3397bf6a: don't follow you there
(10:41:52) shawn-jabber: Let's not worry about that until later. Designing too far ahead ends up making things more complicated than they need to be
(10:41:58) peter.banka@joseywales/3397bf6a: right on
(10:44:38) peter.banka@joseywales/3397bf6a: huh...if we strap this in, we could be buildbot
(10:44:55) peter.banka@joseywales/3397bf6a: but again, that's solving a problem we don't have right now
(10:45:02) shawn-jabber: Sure, I just mean they might have use for an extension of buildbot or somethign

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
Peter Banka
Direction:
Needs approval
Assignee:
Peter Banka
Definition:
Approved
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.