Add support for __spec__

[hauntsaninja]

Fixes #4145

It was too annoying to get the fixtures to work out here, so I didn't. But you can see the tests will add the None for the non __main__ case

Co-authored-by: Joongi Kim me@daybreaker.info

[JukkaL]

I wonder if this may down slow down tests significantly, since every module top level will have to be processed many times? Maybe instead check options.use_builtins_fixtures and fall back to builtins.object immediately if it's true.

[JukkaL]

Consider also adding a pythoneval test that uses full stubs.

[achimnol]

@hauntsaninja @JukkaL Is there anything that I could help to conclude this PR?
It seems not to be too difficult but I'm just not familiar enough with the mypy development.
Maybe somebody could help me in the PyCon US sprint?

[hauntsaninja]

Yeah, that sounds good. And if you want to experiment before sprints, feel free to post a diff on this PR and I can commit and merge it :-)

[achimnol]

I'd like to, but I need some guidance on writing tests in mypy and internal mechanisms mentioned in the previous review. I'll try a look.

[achimnol]

Maybe the pythoneval test update could be:

diff --git a/test-data/unit/pythoneval.test b/test-data/unit/pythoneval.test
index 0ed3540b6..9a43d9071 100644
--- a/test-data/unit/pythoneval.test
+++ b/test-data/unit/pythoneval.test
@@ -107,14 +107,20 @@ f
 [case testModuleAttributes]
 import math
 import typing
+print(type(__spec__))
 print(math.__name__)
+print(math.__spec__.name)
 print(type(math.__dict__))
 print(type(math.__doc__ or ''))
+print(type(math.__spec__))
 print(math.__class__)
 [out]
+<class 'NoneType'>
+math
 math
 <class 'dict'>
 <class 'str'>
+<class '_frozen_importlib.ModuleSpec'>
 <class 'module'>

 [case testSpecialAttributes]

[achimnol]

Maybe the first review would intend the following changes..?

diff --git a/mypy/semanal.py b/mypy/semanal.py
index 5bafe124b..1158d7f7b 100644
--- a/mypy/semanal.py
+++ b/mypy/semanal.py
@@ -662,14 +662,14 @@ class SemanticAnalyzer(
                     return
                 typ = inst
             elif name == "__spec__":
-                inst: Type | None = self.named_type_or_none("importlib.machinery.ModuleSpec")
+                if self.options.use_builtins_fixtures:
+                    inst = self.named_type_or_none("builtins.object")
+                else:
+                    inst = self.named_type_or_none("importlib.machinery.ModuleSpec")
                 if inst is None:
-                    if self.final_iteration:
-                        inst = self.named_type_or_none("builtins.object")
-                        assert inst is not None, "Cannot find builtins.object"
-                    else:
-                        self.defer()
-                        return
+                    assert not self.final_iteration, "Cannot find builtins.object to add __spec__"
+                    self.defer()
+                    return
                 if file_node.name == "__main__":
                     # https://docs.python.org/3/reference/import.html#main-spec
                     inst = UnionType.make_union([inst, NoneType()])

[hauntsaninja]

Thank you, pushed with some minor changes (I think we still need the final_iteration code)! Btw, I'm in Room 320 :-)

[achimnol]

Thanks!!! 👍🏼

[achimnol]

@hauntsaninja Here is another patch to add:

diff --git a/test-data/unit/fine-grained-inspect.test b/test-data/unit/fine-grained-inspect.test
index f8ce35585..ed89f2f09 100644
--- a/test-data/unit/fine-grained-inspect.test
+++ b/test-data/unit/fine-grained-inspect.test
@@ -236,7 +236,7 @@ class C: ...
 [builtins fixtures/module.pyi]
 [out]
 ==
-{"<pack.bar>": ["C", "__annotations__", "__doc__", "__file__", "__name__", "__package__", "bar", "x"], "ModuleType": ["__file__", "__getattr__"]}
+{"<pack.bar>": ["C", "__annotations__", "__doc__", "__file__", "__name__", "__package__", "__spec__", "bar", "x"], "ModuleType": ["__file__", "__getattr__"]}

 [case testInspectModuleDef]
 # inspect2: --show=definition --include-kind tmp/foo.py:2:1

[achimnol]

I'm not sure what's happening with pydantic though.... 🤔

It seems to be the expected result as now mypy recognizes the __spec__ variable.

[achimnol]

This breaks in Python 3.8... 😥

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pydantic (https://github.com/samuelcolvin/pydantic)
- pydantic/__init__.py:223: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:225: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:226: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:227: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:228: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:229: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:230: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:231: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:232: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:234: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:236: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:237: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:238: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:239: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:240: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:242: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:243: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:245: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:247: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:248: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:249: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:250: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:251: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:252: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:254: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:255: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:256: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:258: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:259: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:260: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:262: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:263: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:265: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:266: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:267: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:268: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:269: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:270: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:271: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:272: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:273: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:274: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:275: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:276: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:277: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:278: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:279: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:280: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:281: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:282: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:283: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:284: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:285: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:286: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:287: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:288: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:290: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:292: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:293: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:294: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:295: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:296: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:297: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:298: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:299: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:300: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:301: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:302: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:303: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:304: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:305: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:306: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:307: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:308: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:309: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:310: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:311: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:312: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:313: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:314: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:315: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:316: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:317: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:318: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:319: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:320: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:321: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:322: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:323: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:324: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:325: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:326: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:327: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:328: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:329: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:330: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:331: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:332: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:333: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:334: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:335: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:336: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:337: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:338: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:339: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:340: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:341: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:342: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:343: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:344: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:345: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:346: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:347: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:348: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:349: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:350: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:352: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:354: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:355: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:356: error: Name "__spec__" is not defined  [name-defined]
- pydantic/__init__.py:358: error: Name "__spec__" is not defined  [name-defined]

... (truncated 139 lines) ...