Creating *.js files anywhere under the project root triggers recompilation in watch mode for big projects having an unresolved name

[ogvolkov]

TypeScript Version: 2.7.0-dev.20171230
Code
The issue happens if the project has a name that failed to resolve, and has more than 256 modules. The full demo project reproducing this behavior is available in this repo.

The important parts are as follows.

We would like to import third parties in the same way as normal TypeScript modules. For example, we use jquery type definitions and import it like this:

import * as $ from 'jquery';
export default function who() { return 5; }

Assume that we have more than 256 modules like the one shown above.

The tsconfig.json is

{
  "compilerOptions": {
    "target": "ES2015",
    "module": "amd",
    "declaration": true,
    "outDir": "build",
    "strict": true
  },
  "include": [
    "src/**/*.ts"
  ]
}

Description:
Run TypeScript in watch mode. Add an arbitrary *.js file anywhere under the project root folder.

Expected behavior:
No compilation happens.

Actual behavior:
File change detected. Starting incremental compilation...

This issue is a combination of several factors from resolutionCache.ts.

First, somehow $ name gets unresolved (even though the code compiles perfectly). For some reason, the compiler starts to watch over potential paths where the resolution might be, see resolveNamesWithLocalCache. As one of the potential locations belongs to the root folder, it effectively starts watching for changes in the root folder, recursively.

Next, when JS file is added to the folder, the watcher is triggered. Normally failedLookupDefaultExtensions is used to filter out irrelevant file changes, but it's a constant, and Extension.Js is always included regardless of the project's decision not to compile js.

Then, invalidateResolutions is called, the decision is made to always update the project, if module names collection is larger than maxNumberOfFilesToIterateForInvalidation.

Therefore, adding a new JavaScript file triggers the recompilation.

[mhegazy]

@sheetalkamat can you please take a look.

[sheetalkamat]

The scenario wherein if resolution has failed and you have more than 256 modules, triggering compilation is by design. It is to ensure that the resolution is done in exact same way as in when you that without watch option. And 256 is a limit (flexible but currently set to 256) to not iterate through modules and find the invalidated resolution, instead of just relying on program creation logic which in many cases has proven to be faster and have same result. (way to verify this is through --traceResolution and you will notice that the resolution does take place, but based on the settings program doesnt include js files) So even though its compiling program again, there wont be any files emitted if there was no change in program because of this file addition. Way to verify that is to use --listEmittedFiles and ensure that files aren't being emitted with this addition.

@ogvolkov Having said that, can you please provide log from tsc --watch --diagnostics to understand why you might be going into state where the resolution fails and to see if there is a bug there.

[ogvolkov]

I do understand the rationale behind limiting the number of files to go through. However, the other points are not that clear, namely, (1) why the name is unresolved in the first place, even if the compilation itself succeeds, and (2) why js files are not short-circuited earlier, as they should not have any effect on a ts-only project.

And I do confirm that the actual recompilation is quick and does not seem to change any files. The adverse effect we are seeing happens in the situation when there is a gulp file which compiles TypeScript first, and then processes the resulting files further, generating a lot of intermediate JS in progress. Basically, incremental compilation messages flood all the output, making it very hard to find a meaningful messages from the other steps of the pipeline. And no doubt they confuse developers a lot.

Below is the output of tsc --watch --diagnostics, but also please note I've linked a sample project which you can use to debug the issue.

The first two "Synchronizing program" outputs happen when I start tsc watch, the third one happened when I've added a new js file.

c:\development\ts-mystery-watch-issue>node_modules.bin\tsc --watch --diagnostics
Synchronizing program
Files: 302
Lines: 29748
Nodes: 142063
Identifiers: 47395
Symbols: 37357
Types: 13609
Memory used: 112324K
I/O read: 0.06s
I/O write: 0.56s
Parse time: 0.61s
Bind time: 0.17s
Check time: 1.13s
Emit time: 0.94s
Total time: 2.86s
11:28:59 PM - Compilation complete. Watching for file changes.

11:29:00 PM - File change detected. Starting incremental compilation...

Synchronizing program
Files: 302
Lines: 29748
Nodes: 142063
Identifiers: 47395
Symbols: 30633
Types: 55
Memory used: 106158K
I/O read: 0.00s
I/O write: 0.00s
Parse time: 0.03s
Bind time: 0.00s
Check time: 0.00s
Emit time: 0.00s
Total time: 0.03s
11:29:00 PM - Compilation complete. Watching for file changes.

11:29:25 PM - File change detected. Starting incremental compilation...

Synchronizing program
Files: 302
Lines: 29748
Nodes: 142063
Identifiers: 47395
Symbols: 30633
Types: 55
Memory used: 99330K
I/O read: 0.00s
I/O write: 0.00s
Parse time: 0.05s
Bind time: 0.00s
Check time: 0.00s
Emit time: 0.00s
Total time: 0.05s
11:29:25 PM - Compilation complete. Watching for file changes.

[sheetalkamat]

(1) why the name is unresolved in the first place, even if the compilation itself succeeds

It is tracking any file additions that precede over the the currently resolved resolution, again instead of watching every single file that is needed, we use directory watcher to figure out if addition to files could potentially result in new program

======== Resolving module 'jquery' from 'c:/temp/projs/ts-mystery-watch-issue/src/module0.ts'. ========
Module resolution kind is not specified, using 'Classic'.
File 'c:/temp/projs/ts-mystery-watch-issue/src/jquery.ts' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/src/jquery.tsx' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/src/jquery.d.ts' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/jquery.ts' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/jquery.tsx' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/jquery.d.ts' does not exist.
File 'c:/temp/projs/jquery.ts' does not exist.
File 'c:/temp/projs/jquery.tsx' does not exist.
File 'c:/temp/projs/jquery.d.ts' does not exist.
File 'c:/temp/jquery.ts' does not exist.
File 'c:/temp/jquery.tsx' does not exist.
File 'c:/temp/jquery.d.ts' does not exist.
File 'c:/jquery.ts' does not exist.
File 'c:/jquery.tsx' does not exist.
File 'c:/jquery.d.ts' does not exist.
Directory 'c:/temp/projs/ts-mystery-watch-issue/src/node_modules' does not exist, skipping all lookups in it.
Found 'package.json' at 'c:/temp/projs/ts-mystery-watch-issue/node_modules/@types/jquery/package.json'.
File 'c:/temp/projs/ts-mystery-watch-issue/node_modules/@types/jquery.d.ts' does not exist.
'package.json' does not have a 'typings' field.
'package.json' does not have a 'types' field.
File 'c:/temp/projs/ts-mystery-watch-issue/node_modules/@types/jquery/index.d.ts' exist - use it as a name resolution result.

(2) why js files are not short-circuited earlier, as they should not have any effect on a ts-only project.

Technically we could do this but we chose to rather to cache this info at resolution level rather than resolutions that could change the program structure. Eg. if you didnt have @types/jquery installed the resolution would have included .js files in it, (in the end program wont use the resolution if it turns out to be .js but from module resolution perspective the .js file does change the resolution.) Having done that we don't have to worry about when and what order resolution will have .js files and if addition of those would impact the resolution and that can be handled at later time.

Module resolution kind is not specified, using 'Classic'.
File 'c:/temp/projs/ts-mystery-watch-issue/src/jquery.ts' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/src/jquery.tsx' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/src/jquery.d.ts' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/jquery.ts' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/jquery.tsx' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/jquery.d.ts' does not exist.
File 'c:/temp/projs/jquery.ts' does not exist.
File 'c:/temp/projs/jquery.tsx' does not exist.
File 'c:/temp/projs/jquery.d.ts' does not exist.
File 'c:/temp/jquery.ts' does not exist.
File 'c:/temp/jquery.tsx' does not exist.
File 'c:/temp/jquery.d.ts' does not exist.
File 'c:/jquery.ts' does not exist.
File 'c:/jquery.tsx' does not exist.
File 'c:/jquery.d.ts' does not exist.
Directory 'c:/temp/projs/ts-mystery-watch-issue/src/node_modules' does not exist, skipping all lookups in it.
Directory 'c:/temp/projs/ts-mystery-watch-issue/node_modules/@types' does not exist, skipping all lookups in it.
Directory 'c:/temp/projs/node_modules' does not exist, skipping all lookups in it.
Directory 'c:/temp/node_modules' does not exist, skipping all lookups in it.
Directory 'c:/node_modules' does not exist, skipping all lookups in it.
File 'c:/temp/projs/ts-mystery-watch-issue/src/jquery.js' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/src/jquery.jsx' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/jquery.js' does not exist.
File 'c:/temp/projs/ts-mystery-watch-issue/jquery.jsx' does not exist.
File 'c:/temp/projs/jquery.js' does not exist.
File 'c:/temp/projs/jquery.jsx' does not exist.
File 'c:/temp/jquery.js' does not exist.
File 'c:/temp/jquery.jsx' does not exist.
File 'c:/jquery.js' does not exist.
File 'c:/jquery.jsx' does not exist.

Note that #20427 did work to clear out console before reporting updated compile status and that would help you with the issue you are facing about not being able to locate useful message.

[ogvolkov]

So, if I understand correctly, even in the case of a successful resolution, you still have to watch for other potential locations because another resolution might appear there, potentially changing the compilation result. That is reasonable. I also understand how having a single recursive watcher might be more efficient than having many more specific ones.

But I'm not convinced about the lack of filtering after a file change is detected. The documentation makes an impression that only TypeScript files should matter (it only talks about .js for the original Node.js resolution method, as a reference). Moreover, "Node" resolution strategy appears to be looking only in (various) node_modules, while in fact even if I put "moduleResolution": "Node" in tsconfig, recompilation is still triggered by the files appearing in other folders.

In my point of view, the actual resolution order is pretty confusing if you compare with what is written in the documentation. I can imagine a situation when typings are missing/incorrect, and the compiler picks up a rogue jquery.js from a totally unexpected location under the root, confusing the hell out of a developer. I think having at least an option of consequential ts-only resolution strategy would be pretty beneficial.

[sheetalkamat]

@ogvolkov Note that filtering on js file is not going to solve your purpose since you are also emitting declaration files.. So the compilation would get triggered by the declaration emit as well. We could tweak this little further to ignore the build outputs and that might help with your issue.