Infer block-pass symbols

[castwide]

ref #777

Example:

array = [1, 2, 3]
array.map(&:to_s) # Inferred type is `Array<String>`

Requirements:

• Chains recognize the block-pass symbol syntax
• Modify Chain and Call to support Block and BlockSymbol links
• Chain#infer resolves types from BlockSymbol and resolves the return type's generics

This is going to involve some non-trivial refactoring of source chains. Call links should be responsible for resolving block returns, so they need direct access to Block, BlockVariable, and BlockSymbol links passed into them. BlockSymbol and BlockVariable should not be included in Call#arguments. There should be a Call#block attribute instead. Resolution of the call's return type can then be moved to Call#resolve instead of Chain#infer.

[apiology]

Thanks for pushing this up--glad to take this one off my list! I'll focus on getting apiology#1 ready to review.

Have you looked at the resolution code there?

Incidentally, if you can give me some rough definitions for 'resolve' 'infer' and 'probe', I'd be happy to send a doc PR for the next person. I think I know but suspect I'll learn something if it comes out in your words.

[castwide]

Thanks for pushing this up--glad to take this one off my list! I'll focus on getting apiology#1 ready to review.

Have you looked at the resolution code there?

No, I hadn't seen it. That'll be important to the test that fails array.map(&.to_s). I came close to climbing down that rabbit hole, so thanks for bringing your work on it to my attention.

In the meantime, I can continue work on block chaining. (I'll try not to make too many bitcoin puns.)

Incidentally, if you can give me some rough definitions for 'resolve' 'infer' and 'probe', I'd be happy to send a doc PR for the next person. I think I know but suspect I'll learn something if it comes out in your words.

Chain#infer is the method that determines a code snippet's type through static analysis of a node chain. Clip#infer depends on Chain#infer. In method and variable pins, #probe uses Clip#infer under the hood; other pins fall back to #typify.

Chain::Link#resolve returns an array of pins that can be probed for a return type. Chain#define calls #resolve on each of its links in order, passing the first resolved pin of the last link to the next link, and returns the last link's results.

Here's an example. SourceChainer parses 'string'.length.abs into a chain with three links: a Literal (String), a Call to length, and a Call to abs. Type inference on the chain works like this:

• Chain#infer calls #define.
• Chain#define calls #resolve on each of its links.
• Link 1 (String) returns an anonymous pin with a String return type.
• Link 2 (length) takes the String pin and searches for method pins named length in the String namespace.
• Link 3 (abs) takes the String#length pin and searches for method pins named abs in the Integer namespace.
• Chain#infer takes the last array of pins and returns the first defined return type it finds. It'll try to #typify the pin first, which checks for documented types. If #typify returns undefined, it tries #probe.

Hope that makes sense.

[apiology]

#794 is ready for review if you want to use bits of it

[castwide]

This PR might have exposed a gap in the generics feature.

The currently failing test involves Array#map:

array = [100] # Correctly inferred as [Array<Integer>]
array.map(&:to_s) # Should be [Array<String>] but inferred as [Array] instead

The Array#map return type's generic is specific to the method, not the class. In RBS, Enumerable#map is defined like this:

def map: [U] () { (Elem arg0) -> U } -> ::Array[U]
       | () -> ::Enumerator[Elem, ::Array[untyped]]

I think The YARD equivalent should look something like this:

# @generic U
# @yieldparam [generic<Elem>] (`@generic Elem` is defined on `Enumerable`)
# @yieldreturn [generic<U>]
# @return [Enumerator<generic<Elem>>, Array<generic<U>>]
def map; end

[apiology]

To be pedantic, "method" generics appear to be defined on the signature-level - though the docs in the RBS gem get a little loose with terminology.

e.g. Enumerable#find:

  def find: () { (Elem) -> boolish } -> Elem?
          | () -> ::Enumerator[Elem, Elem?]
          | [T] (_NotFound[T] ifnone) { (Elem) -> boolish } -> (Elem | T)
          | [T] (_NotFound[T] ifnone) -> ::Enumerator[Elem, Elem | T]

[apiology]

# @generic U
# @yieldparam [generic<Elem>] (`@generic Elem` is defined on `Enumerable`)
# @yieldreturn [generic<U>]
# @return [Enumerator<generic<Elem>>, Array<generic<U>>]
def map; end

I think we really need two signatures like RBS has and to resolve between them based on whether a block is passed in to map(). The former one, with the block, actually does return an Array<generic>:

def map: [U] () { (Elem arg0) -> U } -> ::Array[U]

foo = some_enumerable.map { |s| s.length } # foo is an Array<Integer>

Array<Integer> can be resolved by inferring the return type of the block, then adding that info into the call to Pin::Signature#resolve_generics_from_context for it to resolve the signature.

[apiology]

For the block-less signature:

       | () -> ::Enumerator[Elem, ::Array[untyped]]

Note that this is an Enumerator, which has some special powers above and beyond being an Enumerable. I'm a little fuzzy on it myself, but here's an example use from the rbs docs:

# This allows you to chain Enumerators together.  For example, you can map a
# list's elements to strings containing the index and the element as a string
# via:

puts %w[foo bar baz].map.with_index { |w, i| "#{i}:#{w}" } # Types as Array<String> because .with_index() looks at the block return type, not .map
# => ["0:foo", "1:bar", "2:baz"]

The reason that the second type parameter to Enumerator is untyped in this signature is because we can't know what the user is going to produce in their block at the point where we call .map() in this case.

The second type parameter of Enumerator seems to relate to what the return value of .with_index {} and .each {} are - which isn't typically used by anyone I would think and we don't need to sweat. Enumerator even defaults it to void:

class Enumerator[unchecked out Elem, out Return = void] < Object

[apiology]

In general I wonder if we can remove Pin::Method#parameters and #block entirely and instead force code to look at each signature individually. I haven't tried, so I know that's just big talk right now :). If there are only method-level YARD annotations, we'd just map those into the first Pin::Signature.

But agreed that the combined method-level annotations look could like the above if we want YARD's method-level tags to keep to the model of the union of all signatures tags.

[apiology]

Is this line a no-op? It's not getting assigned to anything

[castwide]

The value of line 174 will get returned from the method, but I'm likely to remove the if condition anyway.

[apiology]

Right on - I'm probably getting lost in indention is all.

[castwide]

@apiology Good points about signatures and enumerators. In YARD, it should probably look more like this:

# @generic U
# @overload map
#   @return [Enumerator<generic<Elem>, generic<U>>
# @overload map(&)
#   @yieldparam [generic<Elem>]
#   @yieldreturn [generic<U>]
#   @return [Array<generic<U>>]
def map; end

In general I wonder if we can remove Pin::Method#parameters and #block entirely and instead force code to look at each signature individually. I haven't tried, so I know that's just big talk right now :). If there are only method-level YARD annotations, we'd just map those into the first Pin::Signature.

Some of that functionality is already there. When method pins are generated from YARD, the first signature is generated from the method-level YARD annotations, and the rest are generated from @overload tags. It might make sense to put the method-level signature last instead; presumably, @overload signatures will have more refined specs and should therefore have higher precedence.

[apiology]

I should probably read up on the Yard docs on overloads and make sure I understand their semantics before I open my mouth on that topic :) My thoughts were mainly around our internal representation for sure.

[castwide]

I should probably read up on the Yard docs on overloads and make sure I understand their semantics before I open my mouth on that topic :) My thoughts were mainly around our internal representation for sure.

Yeah, I probably don't need to worry about making the YARD version work yet. For Enumerable in particular, proper inference based on RBS signatures is definitely more urgent. Imagining the YARD representation helps me wrap my head around the internal representation, but changing @overload processing can be deferred to a separate issue.

[apiology]

Array can be resolved by inferring the return type of the block, then adding that info into the call to Pin::Signature#resolve_generics_from_context for it to resolve the signature.

I took a shot at implementing this in #804 - let me know if this seems at least directionally correct