tahoe-lafs/tahoe-lafs
tahoe-lafs
/
tahoe-lafs
2
0
Fork 0
Code Issues Pull Requests 23 Packages Projects Releases 1 Wiki Activity

Add py38 to CI #732

Closed
chadwhitacre wants to merge 13 commits from 2065.add-py38-to-ci into master
pull from: 2065.add-py38-to-ci
merge into: tahoe-lafs:master
Conversation 12 Commits 13 Files Changed 8 +416 -18
chadwhitacre commented 2020-07-02 11:04:22 +00:00 (Migrated from github.com)
Copy Link

I've got tox barely running under Python 3.8 locally, making a PR so I can start seeing what CI does with it.

(I don't have perms on Trac yet to make a new ticket, so my branch name uses the old mega-ticket 2065.)

Punchlist

I've got tox barely running under Python 3.8 locally, making a PR so I can start seeing what CI does with it. (I don't have perms on Trac yet to make a new ticket, so my branch name uses the old mega-ticket [2065](https://www.tahoe-lafs.org/trac/tahoe-lafs/ticket/2065).) ### Punchlist - [x] get py38 to run to completion - [x] think through `tor_requires`/`i2p_requires` - [x] work around pip 19.0.3-related bug: https://github.com/tahoe-lafs/tahoe-lafs/pull/732#issuecomment-652983675 - [x] make a Trac ticket: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3325 - [ ] fix anything else outside of py38 - [x] add ratchet.py and configure tox.ini to work with it - [x] redo the branch & PR - [ ] satisfy towncrier
chadwhitacre commented 2020-07-02 11:25:42 +00:00 (Migrated from github.com)
Copy Link

Hrm, hack to work around lack of a txi2p release is causing problems. 🤔

Collecting txi2p@ git+ssh://git@github.com/str4d/txi2p@0611b9a86172cb70d2f5e415a88eee9f230590b3#egg=txi2p from git+ssh://git@github.com/str4d/txi2p@0611b9a86172cb70d2f5e415a88eee9f230590b3#egg=txi2p (from tahoe-lafs==1.14.0.post359)
  Cloning ssh://git@github.com/str4d/txi2p (to revision 0611b9a86172cb70d2f5e415a88eee9f230590b3) to /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-Sgktrp/txi2p
  Running command git clone -q ssh://git@github.com/str4d/txi2p /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-Sgktrp/txi2p
  Warning: Permanently added the RSA host key for IP address '192.30.255.112' to the list of known hosts.
  git@github.com: Permission denied (publickey).
  fatal: Could not read from remote repository.

I'm also not immediately seeing the connection between tox config and CI config ... where's py38 at? 🧐

Hrm, hack to work around [lack](https://github.com/str4d/txi2p/issues/10) of a txi2p release is causing [problems](https://github.com/tahoe-lafs/tahoe-lafs/pull/732/checks?check_run_id=830298860#step:8:61). 🤔 ``` Collecting txi2p@ git+ssh://git@github.com/str4d/txi2p@0611b9a86172cb70d2f5e415a88eee9f230590b3#egg=txi2p from git+ssh://git@github.com/str4d/txi2p@0611b9a86172cb70d2f5e415a88eee9f230590b3#egg=txi2p (from tahoe-lafs==1.14.0.post359) Cloning ssh://git@github.com/str4d/txi2p (to revision 0611b9a86172cb70d2f5e415a88eee9f230590b3) to /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-Sgktrp/txi2p Running command git clone -q ssh://git@github.com/str4d/txi2p /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-Sgktrp/txi2p Warning: Permanently added the RSA host key for IP address '192.30.255.112' to the list of known hosts. git@github.com: Permission denied (publickey). fatal: Could not read from remote repository. ``` I'm also not immediately seeing the connection between tox config and CI config ... where's py38 at? 🧐
chadwhitacre commented 2020-07-02 11:50:43 +00:00 (Migrated from github.com)
Copy Link

Here's the first py38 run!

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704285475

Here's the first py38 run! https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704285475
chadwhitacre commented 2020-07-02 11:58:59 +00:00 (Migrated from github.com)
Copy Link

TIL towncrier.

Once I get Trac perms I'll create a proper ticket there, and once I have CI working I'll close this PR and open a fresh one from a new branch with the right ticket number and towncrier will be happy and everyone will be happy and we'll all be happy together.

TIL towncrier. Once I get Trac perms I'll create a proper ticket there, and once I have CI working I'll close this PR and open a fresh one from a new branch with the right ticket number and towncrier will be happy and everyone will be happy and we'll all be happy together.
chadwhitacre commented 2020-07-02 12:44:25 +00:00 (Migrated from github.com)
Copy Link

Interesting bug on Travis with pyinstaller when bumping to Xenial, where the system pip is 19.0.3 vs. ancient 9.0.1 on Trusty. Manifests as no such option: --no-use-pep517. That option was added in 19.1, and our pinned pip is 19.1.1.

Working on Trusty: https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704285470
Broken on Xenial: https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704292628

Hypothesis is that somehow invoking as python -m pip under pyinstaller is finding the system pip vs. the tox-installed pip?

Interesting bug on Travis with pyinstaller when bumping to Xenial, where the system pip is 19.0.3 vs. ancient 9.0.1 on Trusty. Manifests as `no such option: --no-use-pep517`. That option was added in 19.1, and our pinned pip is 19.1.1. Working on Trusty: https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704285470 Broken on Xenial: https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704292628 Hypothesis is that somehow invoking as `python -m pip` under `pyinstaller` is finding the system pip vs. the tox-installed pip?
chadwhitacre commented 2020-07-02 12:46:24 +00:00 (Migrated from github.com)
Copy Link

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704292633 shows expected behavior for py38 tho:

  File "/home/travis/build/tahoe-lafs/tahoe-lafs/.tox/py38/lib/python3.8/site-packages/nevow/stan.py", line 150
    raise NotImplementedError, "Stan slot instances are not iterable."
                             ^
SyntaxError: invalid syntax

I.e., the port to Python 3 is blocked on https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3244.

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704292633 shows expected behavior for py38 tho: ``` File "/home/travis/build/tahoe-lafs/tahoe-lafs/.tox/py38/lib/python3.8/site-packages/nevow/stan.py", line 150 raise NotImplementedError, "Stan slot instances are not iterable." ^ SyntaxError: invalid syntax ``` I.e., the port to Python 3 is blocked on https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3244.
exarkun commented 2020-07-02 15:03:03 +00:00 (Migrated from github.com)
Copy Link

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704292633 shows expected behavior for py38 tho:

  File "/home/travis/build/tahoe-lafs/tahoe-lafs/.tox/py38/lib/python3.8/site-packages/nevow/stan.py", line 150
    raise NotImplementedError, "Stan slot instances are not iterable."
                             ^
SyntaxError: invalid syntax

I.e., the port to Python 3 is blocked on https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3244.

Perhaps we could make the tahoe --version step optional/allowed to fail or move it to after the test suite. We suppose it will still be a while before the Nevow dependency is completely removed but I hope there is a sizable chunk of the codebase and test suite which can be ported and run even while the Nevow dependency lingers in the web code. (As I said though, "I hope".)

> https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704292633 shows expected behavior for py38 tho: > > ``` > File "/home/travis/build/tahoe-lafs/tahoe-lafs/.tox/py38/lib/python3.8/site-packages/nevow/stan.py", line 150 > raise NotImplementedError, "Stan slot instances are not iterable." > ^ > SyntaxError: invalid syntax > ``` > > I.e., the port to Python 3 is blocked on https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3244. Perhaps we could make the `tahoe --version` step optional/allowed to fail or move it to after the test suite. We suppose it will still be a while before the Nevow dependency is completely removed but I _hope_ there is a sizable chunk of the codebase and test suite which can be ported and run even while the Nevow dependency lingers in the web code. (As I said though, "I hope".)
wearpants commented 2020-07-02 20:42:21 +00:00 (Migrated from github.com)
Copy Link

I hope there is a sizable chunk of the codebase and test suite which can be ported and run even while the Nevow dependency lingers in the web code. (As I said though, "I hope".)

That's going to be essential, as it sounds like Nevow removal is going to be several more weeks...

> I hope there is a sizable chunk of the codebase and test suite which can be ported and run even while the Nevow dependency lingers in the web code. (As I said though, "I hope".) That's going to be essential, as it sounds like Nevow removal is going to be several more weeks...
👍 1
chadwhitacre commented 2020-07-03 12:06:07 +00:00 (Migrated from github.com)
Copy Link

💃

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704631138

Ran 93 tests in 2.198s
FAILED (expectedFailures=1, failures=1, errors=84, successes=7)
💃 https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704631138 ``` Ran 93 tests in 2.198s FAILED (expectedFailures=1, failures=1, errors=84, successes=7) ```
chadwhitacre commented 2020-07-03 13:27:23 +00:00 (Migrated from github.com)
Copy Link

Debugging this --no-use-pep517 issue ...

Working on Trusty

[5076] /home/travis/build/tahoe-lafs/tahoe-lafs/.tox$ /home/travis/virtualenv/python2.7.14/bin/python -m virtualenv --no-download --python /home/travis/virtualenv/python2.7.14/bin/python pyinstaller
created virtual environment CPython2.7.14.final.0-64 in 428888ms
  creator CPython2Posix(dest=/home/travis/build/tahoe-lafs/tahoe-lafs/.tox/pyinstaller, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/travis/.local/share/virtualenv)
    added seed packages: pip==20.1.1, setuptools==44.1.1, wheel==0.34.2
  activators BashActivator,CShellActivator,PythonActivator,FishActivator,PowerShellActivator

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704656231#L1764-L1769

Broken on Xenial

[4552] /home/travis/build/tahoe-lafs/tahoe-lafs/.tox$ /home/travis/virtualenv/python2.7.15/bin/python -m virtualenv --no-download --python /home/travis/virtualenv/python2.7.15/bin/python pyinstaller
Already using interpreter /home/travis/virtualenv/python2.7.15/bin/python
Using real prefix '/opt/python/2.7.15'
New python executable in /home/travis/build/tahoe-lafs/tahoe-lafs/.tox/pyinstaller/bin/python
Overwriting /home/travis/build/tahoe-lafs/tahoe-lafs/.tox/pyinstaller/lib/python2.7/distutils/__init__.py with new content
Installing setuptools, pip, wheel...
done.

https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704654720#L1504-L1510

Debugging this `--no-use-pep517` issue ... # Working on Trusty ``` [5076] /home/travis/build/tahoe-lafs/tahoe-lafs/.tox$ /home/travis/virtualenv/python2.7.14/bin/python -m virtualenv --no-download --python /home/travis/virtualenv/python2.7.14/bin/python pyinstaller created virtual environment CPython2.7.14.final.0-64 in 428888ms creator CPython2Posix(dest=/home/travis/build/tahoe-lafs/tahoe-lafs/.tox/pyinstaller, clear=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/travis/.local/share/virtualenv) added seed packages: pip==20.1.1, setuptools==44.1.1, wheel==0.34.2 activators BashActivator,CShellActivator,PythonActivator,FishActivator,PowerShellActivator ``` https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704656231#L1764-L1769 # Broken on Xenial ``` [4552] /home/travis/build/tahoe-lafs/tahoe-lafs/.tox$ /home/travis/virtualenv/python2.7.15/bin/python -m virtualenv --no-download --python /home/travis/virtualenv/python2.7.15/bin/python pyinstaller Already using interpreter /home/travis/virtualenv/python2.7.15/bin/python Using real prefix '/opt/python/2.7.15' New python executable in /home/travis/build/tahoe-lafs/tahoe-lafs/.tox/pyinstaller/bin/python Overwriting /home/travis/build/tahoe-lafs/tahoe-lafs/.tox/pyinstaller/lib/python2.7/distutils/__init__.py with new content Installing setuptools, pip, wheel... done. ``` https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/jobs/704654720#L1504-L1510
chadwhitacre commented 2020-07-03 13:35:40 +00:00 (Migrated from github.com)
Copy Link

Okay, I think I'm on the right track:

https://discuss.python.org/t/how-should-virtualenv-behave-respective-of-seed-packages-e-g-pip-by-default/4146

Okay, I think I'm on the right track: https://discuss.python.org/t/how-should-virtualenv-behave-respective-of-seed-packages-e-g-pip-by-default/4146
chadwhitacre commented 2020-07-03 14:10:39 +00:00 (Migrated from github.com)
Copy Link

Broken on Xenial, virtualenv is 16.1.0.
Working on Trusty, virtualenv is 20.0.25.

Broken on Xenial, virtualenv is [16.1.0](https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/builds/704674754#L353). Working on Trusty, virtualenv is [20.0.25](https://travis-ci.org/github/tahoe-lafs/tahoe-lafs/builds/704677045#L649).
exarkun (Migrated from github.com) reviewed 2020-07-03 14:40:26 +00:00
.travis.yml
exarkun (Migrated from github.com) commented 2020-07-03 14:40:26 +00:00
Copy Link

Feel free to drop Python 3.5 from this config, too. That was from an earlier failed porting effort. We don't need to keep trying to support it.

Feel free to drop Python 3.5 from this config, too. That was from an earlier failed porting effort. We don't need to keep trying to support it.
chadwhitacre commented 2020-07-03 21:50:07 +00:00 (Migrated from github.com)
Copy Link

Moving to #734.

Moving to #734.

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
Benchmarking and Performance

   
HTTP Storage Protocol

   
Nevow Removal

   
Python 3 Porting

   
not-for-merge
No Label
Benchmarking and Performance
HTTP Storage Protocol
Nevow Removal
Python 3 Porting
not-for-merge
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/tahoe-lafs#732
No description provided.
Delete Branch "2065.add-py38-to-ci"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?