Add mypy support #915
No reviewers
Labels
No Label
Benchmarking and Performance
HTTP Storage Protocol
Nevow Removal
Python 3 Porting
not-for-merge
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: tahoe-lafs/tahoe-lafs#915
Loading…
Reference in New Issue
No description provided.
Delete Branch "3399.mypy"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3399
This test run demonstrates the checks now passing in CI.
Thanks. Left a bunch of comments inline. Some additional thoughts:
Like the other images we depend on, we should build this one as part of our CI and push it to the account indicated by
DOCKERHUB_CONTEXT
.Also, in a perfect world, this would be a NixOS-based image, and I'd be happy to help with setting it up if you want.
@ -18,1 +17,4 @@
# type ignored as it fails in CI
# (https://app.circleci.com/pipelines/github/tahoe-lafs/tahoe-lafs/1647/workflows/60ae95d4-abe8-492c-8a03-1ad3b9e42ed3/jobs/40972)
from allmydata._version import __version__ # type: ignore
except ImportError:
I deleted this comment and the one below because I was curious what problem mypy reported without them but when I ran mypy it didn't report any problem.
@ -58,2 +58,4 @@
return defer.succeed((shares, desired_share_ids))
def encode_proposal(self, data, desired_share_ids=None):
raise NotImplementedError()
There seem to be no callers of
encode_proposal
in the codebase. On the other hand, the method above this one,encode
, has the same signature asencode_proposal
and seems to have some callers (egsrc/allmydata/immutable/encode.py
makes aCSREncoder
and then calls itsencode
method. So I think someone forgot the method name between writing the interface and implementing.Maybe better to rename it on the interface since all of the real implementation thinks it has the other name.
I guess this pattern could instead be
precondition(accountfile is None or isinstance(accountfile, unicode), accountfile)
.This seems slightly better to me, considering TPTB have decided to eliminate
types.NoneType
but it's also a larger change from what the code was before so if you don't feel like changing it further that's fine with me too.I can find no callers of
IEncoder.set_size
so I think the better option is to delete the method from the interface. Adding force to this argument is that the docstring forIEncoder.set_size
claims that it must be called beforeget_serialized_params
which seems to belong toICodecEncoder
(rather thanIEncoder
) though the docstring doesn't mention this. However,ICodecEncoder
has no similarset_size
method and gets its size information from__init__
(which is outside of any explicitly declared interface).Can you add a link to https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3561 here?
Neither of these methods has any callers. I think we can delete them from the interface instead.
The implementation doesn't actually work with
signing_key=None
anymore. It probably did long ago. Let's take the default off of the interface (and delete the part of the docstring that claims it is allowed) instead of adding it here.@ -566,3 +566,3 @@
def upload(self, new_contents, servermap):
def upload(self, new_contents, servermap, progress=None):
"""
I wonder why mypy doesn't complain about
3eb975748a/src/allmydata/blacklist.py (L152)
I deleted this comment and didn't see any new mypy errors.
I can't find any callers of this
get_servermap
. Can you link to https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3562 from here?This seems unused, can we get away with just deleting it?
Can this be some
Parameters
typing thing similar toSubCommands
?and here
Since these two modules clearly each do have a
dispatch
attribute I wonder why mypy complains that they don't.Wouldn't this be better if it were actually a float instead of an int pretending to be a float? That is,
POLL_INTERVAL = 60.0
?Oh huh. Interesting. I guess this also has to do with foolscap#78? Because it's about mypy thinking that
RIStorageServer.slot_testv_and_readv_and_writev
is the "supertype" definition ofStorageServer.slot_testv_and_readv_and_writev
but it's actually the supertype definition ofStorageServer.remote_slot_testv_and_readv_and_writev
.Can you link to the foolscap ticket here too, or if you think it's not actually going to be resolved when that ticket is fixed, link to some other ticket describing this particular issue?
File and link to a ticket for writing those stubs?
That sounds like an awesome one but I can't reproduce it locally.
@ -47,8 +47,9 @@ class RIDummy(RemoteInterface):
"""
File and link to a ticket for this?
rather, incomplete implementation, right?
Also cannot reproduce this one
It looks like all of the subclasses of
_DirectoryBaseURI
implement these two so I suppose this is fine. I don't know if this is the right path or if it's better to move theimplements
decoration down to the subclasses.I'm not sure I know what this means. Are there some mypy or mypy-zope docs I should read? Neither the docstring for
typing.Type
nor the information on https://realpython.com/python-type-checking/ seems to explain what's happening here.Shoobx/mypy-zope#26 is closed now. It sounds like there is some other resolution required here, perhaps having to do with using some type stubs for Twisted? Alternatively, if you want to move
IToken
toallmydata.interfaces
and that helps then feel free.Either way, please update the comment to explain the pitfall and link to any outstanding tickets for work that might need to be done to address it.
maybe mention it is ignored because it doesn't exist on Python 3? I assume this (and other Python 2/3 ignores) will go away as the porting effort reaches this code. It might be nice to help the porter who gets to such code remember to remove the ignore somehow.
I think I'd like to see both of these merged to their respective masters before we merge this PR, fwiw.
Agreed. There's nothing special about this image except that it has tox on a late Python 3. Probably the existing images can work - they might need some tweaks to support this. I was hoping someone familiar with those images could figure out the recipe after I had something stable on jaraco/multipy-tox, so yes, I'd appreciate the help.
I definitely share that preference. I saw it as a non-blocker, as the checks can have their benefit without the patches, but I would definitely like to see these merged, and making it a blocker will incentivize getting those merged and released.
@ -18,1 +17,4 @@
# type ignored as it fails in CI
# (https://app.circleci.com/pipelines/github/tahoe-lafs/tahoe-lafs/1647/workflows/60ae95d4-abe8-492c-8a03-1ad3b9e42ed3/jobs/40972)
from allmydata._version import __version__ # type: ignore
except ImportError:
Hmm. Perhaps mypy improved these cases. I searched and didn't find anything about it. I tried and can't find a way to replicate whatever was happening, so I'll revert this change.
@ -18,1 +17,4 @@
# type ignored as it fails in CI
# (https://app.circleci.com/pipelines/github/tahoe-lafs/tahoe-lafs/1647/workflows/60ae95d4-abe8-492c-8a03-1ad3b9e42ed3/jobs/40972)
from allmydata._version import __version__ # type: ignore
except ImportError:
Removed in
950ca1893
I definitely aim to minimize the impact on behavior, so I'll stick with this change for now, but I agree it may be preferable to use the idiomatic form separately.
Right. I think what I was trying to say (succinctly enough to fit within the line length limit) was that the interface of this class is incomplete with respect to its declared interface. Fixed in
99da74fff
.Indeed. I don't know either. In other cases, I went the other route. It's a little annoying that every subclass would need to have redundant declarations (interface + subclass), but that's probably a better approach than implementing/not-implementing the interface in the parent class (as seen here).
I don't have any good references. I've just been picking it up by trial-and-error. What I think this is saying is the
INNER_URI_CLASS
is aType
(aka class) that's an instance of the metaclassIVerifierURI
. That's only really a guess, though. I admit, this approach may not be correct, but it's one solution that unblocks the errors that occur when subclasses of DirectoryURIVerifier indicate a different type for INNER_URI_CLASS. The intention here is to declare some type that's common among all DirectoryURIVerifiers.That's right - having type stubs for Twisted is the proper fix. Clarified in
51b0b201b
.Fixed in
efd0aef28
.@ -58,2 +58,4 @@
return defer.succeed((shares, desired_share_ids))
def encode_proposal(self, data, desired_share_ids=None):
raise NotImplementedError()
It looks like
encode_proposal
is a proposed but not-yet-implemented replacement forencode
. See the docstrings in interfaces.py, especially these:d19b1cfd68/src/allmydata/interfaces.py (L1706-L1716)
d19b1cfd68/src/allmydata/interfaces.py (L1724-L1726)
Renaming the method name in the interface isn't possible as the interface definition for encode also exists and provides a different interface.
I believe this implementation as proposed is the cleanest given the proposal has apparently not been accepted. I would recommend removing this proposed interface, replacing it with an issue or PR to capture the work in progress, but independent of this PR.
Fixed in
ea0c10ef8
.Fixed in
d051791e9
.Fixed in
090031cbf
.Fixed in
0e248cb4e
.Here's what I see:
Fixed in
c2d2aba83
.Sure. I'd observed the same, but gave the code the benefit of the doubt and left it around in case it was used through some mysterious usage I couldn't detect. Fixed in
189608e11
.Great idea. Fixed in
602a06e5c
.602a06e5c
I wondered the same thing. I suspect it's because
(create_node, stats_gatherer)
resolves toTuple[ModuleType]
whereModuleType
is the common interface of thecreate_node
andstats_gatherer
modules andModuleType
doesn't generally have adispatch
attribute. My guess is one would need to create an interface. to satisfy this check, and that's not simple because it would also require creating a type for the dispatch property as well.This was surprising to me too. I learned that
int
is a subtype offloat
, so if you want to declare a number that can be an integer or float, you just declare it as a float. I would recommend against adding.0
or.00
. Integers are acceptable here and should be used where convenient and appropriate.Thanks for the insightful review. I've addressed some of the concerns but have many more to go. I'm taking a break and will resume later.
Pull requests:
Fixed in
6b6b8f837
. PTAL.@ -18,1 +17,4 @@
# type ignored as it fails in CI
# (https://app.circleci.com/pipelines/github/tahoe-lafs/tahoe-lafs/1647/workflows/60ae95d4-abe8-492c-8a03-1ad3b9e42ed3/jobs/40972)
from allmydata._version import __version__ # type: ignore
except ImportError:
Turns out the tests fail in CI but not locally. Not sure why the difference. Regardless, I'll add a comment.
@ -18,1 +17,4 @@
# type ignored as it fails in CI
# (https://app.circleci.com/pipelines/github/tahoe-lafs/tahoe-lafs/1647/workflows/60ae95d4-abe8-492c-8a03-1ad3b9e42ed3/jobs/40972)
from allmydata._version import __version__ # type: ignore
except ImportError:
Brought back in
b65ef3cee
with a comment showing where it fails.That commit was insufficient. Amended with
ab2c544efc
.@ -47,8 +47,9 @@ class RIDummy(RemoteInterface):
"""
In
dacdf7f12
, I added a link to the existing upstream issue.I agree, it probably does have to do with that issue. I think I'd missed that detail. Fixed in
5396f9f97
.Done in
1bf71fd69
.Here's what I see:
I don't understand why the issue exists on some platforms but not others. Probably we need an issue to track this, since it's not consistent.
Addressed in
01147f462
.Addressed in
01147f462
.@ -566,3 +566,3 @@
def upload(self, new_contents, servermap):
def upload(self, new_contents, servermap, progress=None):
"""
blacklist.ProhibitedNode
implementsIFileNode
, which doesn't specify anupload
method. On the other hand,mutable.filenode.MutableFileNode
implementsIMutableFileNode
, which does stipulate the interface.In
574613a89
, I've merged with master and resolved the conflicts. Doing so revealed 8 new errors:The five commits through
652222116
address those errors.This is addressed (for now).
With
4a9d3bde5
, I believe this is close, if not ready.I'm not familiar with "ratchet-style job", but my expectation is that there may be some incremental improvements that over time require some maintenance, but overall they should be small. You may be right about there being some maintenance required if foolscap or twisted were to supply stubs, but my instinct is that we can pin on old versions or otherwise degrade the stubs on those projects to maintain status quo if needed.
I believe I've largely if not fully addressed all of the concerns above to the extent that I can (notably, merging PRs in foolscap and mypy-zope are out of my control). Please take another look.
Codecov Report
37% <0%> (ø)
97% <50%> (-2%)
73% <50%> (-2%)
97% <50%> (-1%)
97% <50%> (-1%)
86% <50%> (-<1%)
94% <50%> (-3%)
95% <60%> (-1%)
73% <60%> (-1%)
81% <60%> (-3%)
Continue to review full report at Codecov.
Pushed a change that switches this to one of our Ubuntu images that has Python 3 and tox. Here's an example CI run:
https://app.circleci.com/pipelines/github/LeastAuthority/tahoe-lafs/154/workflows/2dd439d9-514a-4cd3-bc8d-2ccbe6ea8cb9/jobs/2485
Change is
a01078ddec
. Looks good! Thanks for doing that.Now that those are merged and
c7a4cdb44d
is committed, the project now relies on the main branch of the canonical repo.Now that the blockers are cleared, I recommend to merge this change, and defer any emergent loose ends to follow-up efforts.
Thanks! Looks good. Merging.