Dependency analysis in Scripted

Engineering
Kris De Volder
November 20, 2012

Scripted, a JavaScript editor from VMWare was announced on this blog last month. In this article we’ll take a look under the hood at Scripted’s Dependency Analysis Engine. But before diving into the details, lets motivate why we need it.
Main Motivation: Cross-file Content Assist

To provide a great JavaScript editing experience, Scripted needs to provide accurate suggestions about the functions, methods or identifiers you can use in your current editor context.

[caption id=“attachment_12178” align=“aligncenter” width=“533” caption=“Cross-file Content Assist”][/caption]

Two components work together to achieve this goal:

a fine-grained type inference analysis enginea coarse-grained dependency analysis engine

The inference engine parses your code and walks every declaration, statement and expression. This allows it to determine what identifiers are valid in a given context, and make good guesses about the kinds of things that may be stored in these variables. This information is then used to make content assist suggestions.

If you wanted to simply put all your code into one big file, then a good quality inferencer alone would be sufficient to provide some pretty good content assist. In reality, projects will be divided into modules. In the example above, the ‘main’ module imports a ‘utils’ module. While you edit the main module, Scripted proposes functions from ‘utils’ as appropriate. The dependency analysis engine is what makes this possible.
Other Uses of Dependency Analysis

Scripted also uses dependency analysis to flag unresolved dependencies as errors. For example, if we try to import a ‘bogus’ module Scripted will show an error marker.

[caption id=“attachment_12180” align=“aligncenter” width=“518” caption=“Error Marker for an Unresolved Module”][/caption]

Scripted also uses dependency analysis to support easy navigation. A shift or ctrl click on a resolved dependency name will take you to the corresponding file.

In the future, dependency analysis might also allow us to implement refactoring tools. For example, if you drag-and-drop a .js file to a different directory, Scripted could automatically update relative path references as needed.
What does it do?

The dependency analysis engine provides just a single getDGraph function to its client, the type inference engine:

getDGraph :: ->

This function computes a JSON representation of the dependency graph. This graph contains a node for all files that the target file directly or transitively depends on. If we pass it our ‘main’ module, it will return something like the following:

getDGraph(‘/sample/main.js’) ==>
{
…
“/NODE-NATIVES/stream.js”: {
“kind”: “commonjs”,
“refs”: { … }
},
“/NODE-NATIVES/fs.js”: {
“kind”: “commonjs”,
“refs”: {
“stream”: {
“kind”: “commonjs”,
“name”: “stream”,
“path”: “/NODE-NATIVES/stream.js”
},
…
}
},
“/sample/utils.js”: {
“kind”: “commonjs”,
“refs”: {}
},
“/sample/main.js”: {
“kind”: “commonjs”,
“refs”: {
“fs”: {
“kind”: “commonjs”,
“name”: “fs”,
“path”: “/NODE-NATIVES/fs.js”
},
“./utils”: {
“kind”: “commonjs”,
“name”: “./utils”,
“path”: “/sample/utils.js”
}
}
}
}

Each property in this JSON object represents a node in the graph. The ‘refs’ property contains the edges. Each edge corresponds to a module import.

An interesting detail is that the dependency analyzer returns special path strings for native node modules. When the inference engine request the source code for such a path, the Scripted server, which is written in JavaScript and running on Node.js, extracts the source code from its very own Node.js process. The inference engine analyzes it just like an ordinary JavaScript file. The result is nice content assist for built-in node modules:

[caption id=“attachment_12184” align=“aligncenter” width=“587” caption=“Inferred Suggestions for Built-in Node Modules”][/caption]
What module systems are supported?

For now we only provide support for AMD and CommonJS. For CommonJS we use the enhanced-resolve library to resolve references to paths. For AMD we currently use a custom-made resolver. We may replace this with an existing library if we find one that meets our needs.
How does it work?

The process starts with the target file (i.e. the argument passed to the getDGraph function). It proceeds roughly in the following steps:

Detect the module type (CommonJS versus AMD).Find references in the target module.Resolve the references (i.e. determine the path of the actual file that will get loaded for a reference).Repeat the process from step 1 for each of the resolved references.

Step 1 is based on detecting some typical code patterns. E.g. “a call to ‘require’ not nested inside of a ‘define’ call” is a sign that we are dealing with a CommonJS module.

Steps 2 and 3 are dispatched to different support modules depending on the module type from step 1. It should be relatively easy to add support for additional module types (provided the step 1 detector can be made to recognize the new module type).
Automatic Resolver Configuration

The dependency analyzer tries to discover what it needs to know rather than requiring manual configuration. This is a general philosophy in Scripted and the dependency analyzer is no exception. In fact, there is no way to manually configure the dependency analyzer or any of its components, though it is likely we will make this possible in the future.

For Node/CommonJS this works very well, mainly because there really is not much that needs to be configured. I.e. if we assume the standard Node.js loader algorithm is used, that is really all the information we need.

For AMD the situation is unfortunately more complicated. The typical AMD loader (e.g. requirejs) is highly configurable. Moreover the way this configuration is expressed in the project’s source code tends to vary from project to project. This makes it a challenge to determine where to find the required information in a random project.

The approach we have taken is to look at some example projects and the ‘typical’ patterns they use. Discovery works by recognizing these patterns. The hope is that if we support enough common patterns, eventually discovery will work for most projects. We may also add some manual configuration options as a last resort, for those cases where it fails.

To give an idea how AMD discovery works, here’s an example of one of the patterns we currently detect:

The pattern here is a script tag with a data-main attribute… If Scripted finds this, it will go looking for a requirejs style configuration block in the data-main file:

AMD configuration discovery is a work in progress. As we are presented with more examples of how people setup their AMD loaders we try to add detectors for them. If Scripted incorrectly flags many dependencies as errors, it probably failed to discover your AMD configuration. You can help us by raising a bug request. If you attach a code sample illustrating your ‘typical pattern’. This will help us extend our ‘pattern catalog’.
Conclusion

We took a look under the covers of Scripted. We presented the Scripted dependency-analysis engine. It currently understands both AMD and CommonJS module systems. Dependency analysis provides the type inference engine with a dependency graph. Cross-file analysis based on this graph allows us to make accurate content assist suggestions for functionality defined in other modules. The dependency information is also used to create error markers for unresolvable module references, and allow navigation quick navigation to resolved dependencies. In the future, dependency analysis may also enable the implementation of accurate refactoring tools to move or rename modules.

Want to try out Scripted? Grab it from GitHub. It is easy to install. Download and installation instructions are in the readme file.
Links

ScriptedGitHub (issue tracker, wiki etc.): https://github.com/scripted-editor/scripted/Google Group: https://groups.google.com/forum/#!forum/scripted-devInformation on JavaScript Module Systems and loaders:RequireJS the ‘typical’ AMD module loader: http://requirejs.org/Information on the Node.js module loader http://nodejs.org/docs/latest/api/modules.htmlAMD or ‘Asycnhronous Module Definition’: https://github.com/amdjs/amdjs-api/wiki/AMDCommonJS: http://www.commonjs.org/

comments powered by Disqus

translate：
翻译：

上个月，一个来自VMWare的JavaScript编辑器在这个博客上发布。在本文中，我们将深入了解Scripted的依赖性分析引擎。但在深入讨论细节之前，让我们先来解释一下为什么我们需要它。
主要动机：跨文件内容辅助
为了提供良好的JavaScript编辑体验，Scripted需要提供关于可以在当前编辑器上下文中使用的函数、方法或标识符的准确建议。

Dependency analysis in Scripted相关推荐

(论文笔记06.High Fidelity Data Reduction for Big Data Security Dependency Analyses(CCF A)2016)
High Fidelity Data Reduction for Big Data Security Dependency Analyses(CCF A) 这是我读的条理最清晰的一篇文章了! 1.AB ...
Scons构建C++项目
旧博文,搬到 csdn 原文:http://rebootcat.com/2020/08/30/scons/ 前言我是一个 linux c++ 开发者,但是一直对 Makefile 的语法很是头痛,每 ...
关系抽取论文总结（relation extraction）不断更新
2000 1.Miller, Scott, et al. "A novel use of statistical parsing to extract information from te ...
启动VIP报CRS-1028/CRS-0223致使VIP状态为UNKNOWN故障分析与解决
CRS版本号为10.2.0.4 1.VIP State为UNKNOWN [root@XXdb1 ~]# crs_stat -t Name Type Target ...
天津大学计算机学院院长及副院长,李晓红教授
个人简介李晓红,教授,博士生导师,智能与计算学部网络安全学院副院长.软件与信息安全工程研究所所长,天津市先进网络重点室副主任.兼任计算机学会高级会员.中国计算机学会女工委委员.ACM会员.软件工程专 ...
SCons: A software construction tool
SCons: A software construction tool What is SCons? SCons is an Open Source software construction too ...
ASP.NET Core使用Jaeger实现分布式追踪
前言最近我们公司的部分.NET Core的项目接入了Jaeger,也算是稍微完善了一下.NET团队的技术栈. 至于为什么选择Jaeger而不是Skywalking,这个问题我只能回答,大佬们说了算. ...
java jigsaw_Java 9和Project Jigsaw如何破坏您的代码
java jigsaw Java 9迫在眉睫,它将带有完整的Project Jigsaw . 在我从OpenJFX邮件列表的最近讨论中得知它可能会破坏现有代码之前,我并没有对此给予太多关注. 这对于J ...
Java 9和Project Jigsaw如何破坏您的代码
Java 9迫在眉睫,它将带有完整的Project Jigsaw . 在我从OpenJFX邮件列表的最近讨论中得知它可能会破坏现有代码之前,我并没有对此给予太多关注. 这对于Java非常不寻常,因此引 ...

Dependency analysis in Scripted

Dependency analysis in Scripted相关推荐

最新文章

热门文章