move logic for build steps from buildmaster config to misc/build_helpers #1248

New Issue

tahoe-lafs · 2010-11-04T01:36:41Z

davidsarah commented

2010-11-04 01:36:41 +00:00

There is some near-duplicated code in the buildmaster's TestFromEgg and TestFromPrefixDir classes (in bbsupport.py). There is also duplication between source:setup.py, source:misc/build_helpers/run_trial.py, and some of the tests in source:src/allmydata/test/test_runner.py, for things like setting up PYTHONPATH, getting the Tahoe-LAFS version, and checking that we are running code from the right directory.

Code in the buildmaster is more difficult to debug than code in the Tahoe-LAFS distribution; changes require a buildmaster restart to take effect, and interfere with the reproducibility of builds. So we should simplify these classes by having them just call scripts (or a single script with different arguments) in source:misc/build_helpers.

The reason why the code that is already in the distribution isn't better-factored, is probably that in each case we need to run it before importing any non-stdlib modules. It seems like there should be some way to work around that.

There is some near-duplicated code in the buildmaster's `TestFromEgg` and `TestFromPrefixDir` classes (in `bbsupport.py`). There is also duplication between source:setup.py, source:misc/build_helpers/run_trial.py, and some of the tests in source:src/allmydata/test/test_runner.py, for things like setting up `PYTHONPATH`, getting the Tahoe-LAFS version, and checking that we are running code from the right directory. Code in the buildmaster is more difficult to debug than code in the Tahoe-LAFS distribution; changes require a buildmaster restart to take effect, and interfere with the reproducibility of builds. So we should simplify these classes by having them just call scripts (or a single script with different arguments) in source:misc/build_helpers. The reason why the code that is already in the distribution isn't better-factored, is probably that in each case we need to run it before importing any non-stdlib modules. It seems like there should be some way to work around that.

tahoe-lafs added the

labels 2010-11-04 01:36:41 +00:00

tahoe-lafs added this to the 1.9.0 milestone 2010-11-04 01:36:41 +00:00

davidsarah commented

2010-12-01 19:55:39 +00:00

From #1275, which was a duplicate:

Buildmaster config is a bad place to keep logic—almost nobody can see it except for a privileged few, and it tends to be ill-documented and ill-organized. Move the install-to-prefix step (below) and the test-from-prefix step (below) from buildmaster config into a script in the tahoe-lafs source tree. Also add a test to it that goes red if the install puts anything other than allmydata into the Python lib dir (currently the install puts buildtest into there, which is not intended and will be fixed as soon as we have a test of it).

install-to-prefix:

class InstallToPrefixDir(PythonCommand):
    """
    Step to install into a temporary install directory using --prefix.
    """

    flunkOnFailure = False
    description = ["install-to-prefix"]
    name = "install-to-prefix"

    def __init__(self, prefixinstalldir="prefixinstalldir", *args, **kwargs):
        python_command = ["-c", ("import subprocess, sys;"
                             "sys.exit(subprocess.call([sys.executable, 'setup.py', 'install', '--single-version-externally-managed', '--record=record.txt', '--prefix', '"+prefixinstalldir+"']))")]
        kwargs['python_command'] = python_command
        PythonCommand.__init__(self, *args, **kwargs)
        self.addFactoryArguments(prefixinstalldir=prefixinstalldir)

example: http://tahoe-lafs.org/buildbot/builders/Kyle%20OpenBSD%20amd64/builds/24/steps/install-to-prefix/logs/stdio

test-from-prefix:

class TestFromPrefixDir(PythonCommand):
    """
    Step to run the Tahoe-LAFS tests from a --prefix=installdir installation.
    """
    flunkOnFailure = True
    description = ["test-from-prefixdir"]
    name = "test-from-prefixdir"

    def __init__(self, testsuite=None, prefixinstalldir="prefixinstalldir", srcbasedir=".", *args, **kwargs):
        if testsuite:
            assert isinstance(testsuite, basestring)
            pcmd = (
                    "import copy,os,subprocess,sys;"
                    "trial=os.path.join(os.getcwd(), 'misc', 'build_helpers', 'run_trial.py');"
                    "os.chdir('"+srcbasedir+"');"
                    "testsuite='"+testsuite+"';"
                    "prefixinstdir=os.path.join(os.getcwd(), 'prefixinstalldir');"
                    "libdir=(('win32' in sys.platform.lower()) and os.path.join(os.getcwd(), 'prefixinstalldir', 'Lib', 'site-packages') or os.path.join(os.getcwd(), 'prefixinstalldir', 'lib', 'python%(ver)s' % {'ver': sys.version[:3]}, 'site-packages'));"
                    "bindir=('win32' in sys.platform.lower()) and 'Scripts' or 'bin';"
                    "env=copy.copy(os.environ);"
                    "env['PATH']=bindir+os.pathsep+env.get('PATH','');"
                    "env['PYTHONPATH']=libdir+os.pathsep+env.get('PYTHONPATH','');"
                    "os.chdir('prefixinstalldir');"
                    "sys.exit(subprocess.call([sys.executable, trial, '--reporter=bwverbose', '--rterror', testsuite], env=env))")
        else:
            pcmd = (
                    "import copy,os,subprocess,sys;"
                    "trial=os.path.join(os.getcwd(), 'misc', 'build_helpers', 'run_trial.py');"
                    "os.chdir('"+srcbasedir+"');"
                    "testsuite=subprocess.Popen([sys.executable, 'setup.py', '--name'], stdout=subprocess.PIPE).communicate()[0].strip()+'.test';"
                    "prefixinstdir=os.path.join(os.getcwd(), 'prefixinstalldir');"
                    "libdir=(('win32' in sys.platform.lower()) and os.path.join(os.getcwd(), 'prefixinstalldir', 'Lib', 'site-packages') or os.path.join(os.getcwd(), 'prefixinstalldir', 'lib', 'python%(ver)s' % {'ver': sys.version[:3]}, 'site-packages'));"
                    "bindir=('win32' in sys.platform.lower()) and 'Scripts' or 'bin';"
                    "env=copy.copy(os.environ);"
                    "env['PATH']=bindir+os.pathsep+env.get('PATH','');"
                    "env['PYTHONPATH']=libdir+os.pathsep+env.get('PYTHONPATH','');"
                    "os.chdir('prefixinstalldir');"
                    "sys.exit(subprocess.call([sys.executable, trial, '--reporter=bwverbose', '--rterror', testsuite], env=env))")
        python_command = ["-c", pcmd]
        logfiles = {"test.log": prefixinstalldir+"/_trial_temp/test.log"}
        kwargs['python_command'] = python_command
        kwargs['logfiles'] = logfiles
        PythonCommand.__init__(self, *args, **kwargs)
        self.addFactoryArguments(testsuite=testsuite, prefixinstalldir=prefixinstalldir, srcbasedir=srcbasedir)

example: http://tahoe-lafs.org/buildbot/builders/Kyle%20OpenBSD%20amd64/builds/24/steps/test-from-prefixdir/logs/stdio

(Note: we also need to copy the in-source-tree buildstep script into zfec and pycryptopp source trees.)

From #1275, which was a duplicate: Buildmaster config is a bad place to keep logic—almost nobody can see it except for a privileged few, and it tends to be ill-documented and ill-organized. Move the install-to-prefix step (below) and the test-from-prefix step (below) from buildmaster config into a script in the tahoe-lafs source tree. Also add a test to it that goes red if the install puts anything other than `allmydata` into the Python lib dir (currently the install puts `buildtest` into there, which is not intended and will be fixed as soon as we have a test of it). install-to-prefix: ``` class InstallToPrefixDir(PythonCommand): """ Step to install into a temporary install directory using --prefix. """ flunkOnFailure = False description = ["install-to-prefix"] name = "install-to-prefix" def __init__(self, prefixinstalldir="prefixinstalldir", *args, **kwargs): python_command = ["-c", ("import subprocess, sys;" "sys.exit(subprocess.call([sys.executable, 'setup.py', 'install', '--single-version-externally-managed', '--record=record.txt', '--prefix', '"+prefixinstalldir+"']))")] kwargs['python_command'] = python_command PythonCommand.__init__(self, *args, **kwargs) self.addFactoryArguments(prefixinstalldir=prefixinstalldir) ``` example: <http://tahoe-lafs.org/buildbot/builders/Kyle%20OpenBSD%20amd64/builds/24/steps/install-to-prefix/logs/stdio> test-from-prefix: ``` class TestFromPrefixDir(PythonCommand): """ Step to run the Tahoe-LAFS tests from a --prefix=installdir installation. """ flunkOnFailure = True description = ["test-from-prefixdir"] name = "test-from-prefixdir" def __init__(self, testsuite=None, prefixinstalldir="prefixinstalldir", srcbasedir=".", *args, **kwargs): if testsuite: assert isinstance(testsuite, basestring) pcmd = ( "import copy,os,subprocess,sys;" "trial=os.path.join(os.getcwd(), 'misc', 'build_helpers', 'run_trial.py');" "os.chdir('"+srcbasedir+"');" "testsuite='"+testsuite+"';" "prefixinstdir=os.path.join(os.getcwd(), 'prefixinstalldir');" "libdir=(('win32' in sys.platform.lower()) and os.path.join(os.getcwd(), 'prefixinstalldir', 'Lib', 'site-packages') or os.path.join(os.getcwd(), 'prefixinstalldir', 'lib', 'python%(ver)s' % {'ver': sys.version[:3]}, 'site-packages'));" "bindir=('win32' in sys.platform.lower()) and 'Scripts' or 'bin';" "env=copy.copy(os.environ);" "env['PATH']=bindir+os.pathsep+env.get('PATH','');" "env['PYTHONPATH']=libdir+os.pathsep+env.get('PYTHONPATH','');" "os.chdir('prefixinstalldir');" "sys.exit(subprocess.call([sys.executable, trial, '--reporter=bwverbose', '--rterror', testsuite], env=env))") else: pcmd = ( "import copy,os,subprocess,sys;" "trial=os.path.join(os.getcwd(), 'misc', 'build_helpers', 'run_trial.py');" "os.chdir('"+srcbasedir+"');" "testsuite=subprocess.Popen([sys.executable, 'setup.py', '--name'], stdout=subprocess.PIPE).communicate()[0].strip()+'.test';" "prefixinstdir=os.path.join(os.getcwd(), 'prefixinstalldir');" "libdir=(('win32' in sys.platform.lower()) and os.path.join(os.getcwd(), 'prefixinstalldir', 'Lib', 'site-packages') or os.path.join(os.getcwd(), 'prefixinstalldir', 'lib', 'python%(ver)s' % {'ver': sys.version[:3]}, 'site-packages'));" "bindir=('win32' in sys.platform.lower()) and 'Scripts' or 'bin';" "env=copy.copy(os.environ);" "env['PATH']=bindir+os.pathsep+env.get('PATH','');" "env['PYTHONPATH']=libdir+os.pathsep+env.get('PYTHONPATH','');" "os.chdir('prefixinstalldir');" "sys.exit(subprocess.call([sys.executable, trial, '--reporter=bwverbose', '--rterror', testsuite], env=env))") python_command = ["-c", pcmd] logfiles = {"test.log": prefixinstalldir+"/_trial_temp/test.log"} kwargs['python_command'] = python_command kwargs['logfiles'] = logfiles PythonCommand.__init__(self, *args, **kwargs) self.addFactoryArguments(testsuite=testsuite, prefixinstalldir=prefixinstalldir, srcbasedir=srcbasedir) ``` example: <http://tahoe-lafs.org/buildbot/builders/Kyle%20OpenBSD%20amd64/builds/24/steps/test-from-prefixdir/logs/stdio> (Note: we also need to copy the in-source-tree buildstep script into zfec and pycryptopp source trees.)

tahoe-lafs added

major

and removed

minor

labels 2010-12-01 19:55:39 +00:00

tahoe-lafs modified the milestone from 1.9.0 to soon

2010-12-01 19:55:39 +00:00

tahoe-lafs changed title from ~~refactor buildmaster Test classes and build_helper scripts~~ to move logic for build steps from buildmaster config to misc/build_helpers

2010-12-01 19:55:39 +00:00

zooko commented

2011-01-14 23:42:02 +00:00

François: want to work on this, this weekend? We could work on it together.

francois commented

2011-01-15 16:21:34 +00:00

Replying to zooko:

François: want to work on this, this weekend? We could work on it together.

My current priority is #755, but I'll be glad to work on this one as soon as #755 is fixed.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/1248#issuecomment-80957): > François: want to work on this, this weekend? We could work on it together. My current priority is #755, but I'll be glad to work on this one as soon as #755 is fixed.

tahoe-lafs added

normal

and removed

major

labels 2012-04-01 05:04:01 +00:00

davidsarah commented

2012-05-31 22:33:43 +00:00

: have I mentioned recently that the buildbot config is way too big, and that things like this should be in a Makefile or a shell script or a helper program inside the source tree, so they can be updated at the same time?

<warner>: have I mentioned recently that the buildbot config is way too big, and that things like this should be in a Makefile or a shell script or a helper program inside the source tree, so they can be updated at the same time?

zooko commented

2013-08-27 12:13:59 +00:00

Duplicate of #2049.

zooko added the

duplicate

label 2013-08-27 12:13:59 +00:00

zooko closed this issue

2013-08-27 12:13:59 +00:00

warner commented

2015-01-13 17:51:29 +00:00

On second thought, #2049 doesn't really cover this (it's about where the packaging tests should live, not about where the subroutines used by those tests should live). We decided to reopen this.

warner removed the

duplicate

label 2015-01-13 17:51:29 +00:00

warner reopened this issue

2015-01-13 17:51:29 +00:00

zooko commented

2015-01-13 18:01:16 +00:00

Here is the set of packaging tests which are currently written into the buildmaster config and which ought to be moved into separate scripts:

Here is the set of packaging tests which are currently written into the buildmaster config and which ought to be moved into separate scripts: * [BuiltTest](https://github.com/tahoe-lafs/buildbot-config-tahoe/blob/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l78) * [TestOldDep](https://github.com/tahoe-lafs/buildbot-config-tahoe/blob/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l124) * [TestAlreadyHaveDep](https://github.com/tahoe-lafs/buildbot-config-tahoe/blob/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l136) * [TestFromEggTrial](https://github.com/tahoe-lafs/buildbot-config-tahoe/blob/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l477) * [TestFromEgg](https://github.com/tahoe-lafs/buildbot-config-tahoe/blob/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l518) * [TestFromPrefixDirPyunit](https://github.com/tahoe-lafs/buildbot-config-tahoe/blob/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l561) * [TestFromPrefixDirRunTrial](https://github.com/tahoe-lafs/buildbot-config-tahoe/blob/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l598)

exarkun commented

2019-07-25 12:59:26 +00:00

Buildbot has been decommissioned.

exarkun added the

wontfix

label 2019-07-25 12:59:26 +00:00

exarkun closed this issue

2019-07-25 12:59:26 +00:00

Sign in to join this conversation.