WIP: Use PMCD returned by mappinganalysis to build minimal graph for query #3488

YannanGao-gs · 2024-08-30T21:14:11Z

Summary

Use PMCD returned by mappinganalysis to build minimal graph for query

clean/update: #2523

How did you test this change?

Test(s) added
Manual testing (please provide screenshots/recordings)
No testing (please provide an explanation)

changeset-bot · 2024-08-30T21:14:15Z

🦋 Changeset detected

Latest commit: 136f682

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 26 packages

Name	Type
@finos/legend-extension-dsl-data-space	Minor
@finos/legend-application-query	Minor
@finos/legend-extension-dsl-data-space-studio	Patch
@finos/legend-extension-dsl-data-quality	Patch
@finos/legend-application-studio	Patch
@finos/legend-query-builder	Patch
@finos/legend-graph	Patch
@finos/legend-application-query-bootstrap	Patch
@finos/legend-application-studio-bootstrap	Patch
@finos/legend-extension-dsl-service	Patch
@finos/legend-extension-assortment	Patch
@finos/legend-extension-dsl-diagram	Patch
@finos/legend-extension-dsl-persistence	Patch
@finos/legend-extension-dsl-text	Patch
@finos/legend-extension-store-flat-data	Patch
@finos/legend-extension-store-relational	Patch
@finos/legend-extension-store-service-store	Patch
@finos/legend-vscode-extension-dependencies	Patch
@finos/legend-application-pure-ide	Patch
@finos/legend-application-repl	Patch
@finos/legend-lego	Patch
@finos/legend-application-query-deployment	Patch
@finos/legend-application-studio-deployment	Patch
@finos/legend-application-pure-ide-deployment	Patch
@finos/legend-application-repl-deployment	Patch
@finos/legend-server-showcase-deployment	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

codecov · 2024-08-30T21:18:28Z

Codecov Report

Attention: Patch coverage is 17.36227% with 990 lines in your changes missing coverage. Please review.

Project coverage is 45.44%. Comparing base (ac48624) to head (136f682).
Report is 19 commits behind head on master.

Files with missing lines	Patch %	Lines
...c/components/Core_LegendQueryApplicationPlugin.tsx	17.37%	214 Missing ⚠️
...e/v1/V1_DSL_DataSpace_PureGraphManagerExtension.ts	4.89%	175 Missing ⚠️
...ph-manager/protocol/pure/v1/V1_PureGraphManager.ts	2.06%	95 Missing ⚠️
...d-application-query/src/stores/QueryEditorStore.ts	21.00%	94 Missing ⚠️
...rc/stores/data-space/DataSpaceQueryCreatorStore.ts	4.54%	84 Missing ⚠️
...s/data-space/DataSpaceTemplateQueryCreatorStore.ts	13.68%	82 Missing ⚠️
...stores/query-builder/DataSpaceQueryBuilderState.ts	0.00%	42 Missing ⚠️
...components/query-builder/DataSpaceQueryBuilder.tsx	0.00%	29 Missing ⚠️
...r/action/analytics/MappingModelCoverageAnalysis.ts	10.34%	26 Missing ⚠️
...gend-query-builder/src/stores/QueryBuilderState.ts	13.04%	20 Missing ⚠️
... and 20 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3488      +/-   ##
==========================================
- Coverage   45.76%   45.44%   -0.32%     
==========================================
  Files        2134     2139       +5     
  Lines      367464   371079    +3615     
  Branches    15810    11497    -4313     
==========================================
+ Hits       168152   168620     +468     
- Misses     198382   201781    +3399     
+ Partials      930      678     -252

Files with missing lines	Coverage Δ
...src/__lib__/DSL_DataSpace_LegendQueryNavigation.ts	`59.84% <100.00%> (+0.64%)`	⬆️
...s/__test-utils__/QueryEditorComponentTestUtils.tsx	`100.00% <100.00%> (ø)`
...-query/src/stores/LegendQueryApplicationPlugin.tsx	`100.00% <100.00%> (ø)`
...r-group/mapping-editor/MappingExecutionBuilder.tsx	`58.50% <100.00%> (ø)`
...ng-editor/legacy/DEPRECATED__MappingTestEditor.tsx	`69.57% <100.00%> (ø)`
...itor/editor-group/uml-editor/ClassQueryBuilder.tsx	`42.11% <100.00%> (ø)`
...dio/src/stores/editor/EmbeddedQueryBuilderState.ts	`79.66% <100.00%> (+0.17%)`	⬆️
...ol/pure/DSL_DataSpace_PureGraphManagerExtension.ts	`91.35% <100.00%> (+2.12%)`	⬆️
...raph/src/graph-manager/AbstractPureGraphManager.ts	`93.82% <100.00%> (+0.10%)`	⬆️
...editor-group/function-activator/FunctionEditor.tsx	`59.48% <0.00%> (ø)`
... and 29 more

... and 239 files with indirect coverage changes

YannanGao-gs · 2024-09-04T13:39:24Z

packages/legend-application-query/src/components/Core_LegendQueryApplicationPlugin.tsx

+    ];
+  }
+
+  getExtraQueryBuilderPropagateExecutionContextChangeHelper?(): QueryBuilderPropagateExecutionContextChangeHelper[] {


HEADS-UP: create an extension method for propagateExecutionContextChange because for dataspace, it depends on editorStore (@finos/legend-application-query) and dataspaceQueryBuilderState (@finos/legend-extension-dsl-data-space).
#2523 put this function's implementaion at module @finos/legend-extension-dsl-data-space, but it will introduce the circular dependency as it can't get editorStore inside this module

YannanGao-gs · 2024-09-04T13:48:04Z

HEADS-UP:

To fix (regressions)

about dataspace will not work due to missing connection (modify result json to have connection path)
dataspace template query is not working due to empty executables
either add diagram analyze coverage in engine or bypass diagram check in secondpass in studio

YannanGao-gs · 2024-09-04T13:58:20Z

packages/legend-application-query/src/stores/data-space/DataSpaceQueryCreatorStore.ts

@@ -182,6 +186,10 @@ export class DataSpaceQueryCreatorStore extends QueryEditorStore {
    super.initialize();
  }

+  override requiresGraphBuilding(): boolean {


why is this needed?

YannanGao-gs · 2024-09-04T14:28:33Z

packages/legend-query-builder/src/stores/workflows/ServiceQueryBuilderState.ts

-      const possibleNewClass = getNullableFirstEntry(compatibleClasses);
-      if (possibleNewClass) {
-        this.changeClass(possibleNewClass);
+  override async propagateExecutionContextChange(


polymorphism in OOD will make sure this specific implementation will be triggered

YannanGao-gs · 2024-09-04T14:38:49Z

...-data-space/src/graph-manager/protocol/pure/v1/V1_DSL_DataSpace_PureGraphManagerExtension.ts

+    ) {
+      analysisResult = cacheResult;
+    } else {
+      notificationService?.notify(


HEADS-UP: I moved this notification inside the else block from where it was originally added above the if statement

YannanGao-gs · 2024-09-04T16:45:16Z

packages/legend-extension-dsl-data-space/src/stores/query-builder/DataSpaceQueryBuilderState.ts

@@ -362,7 +384,8 @@ export class DataSpaceQueryBuilderState extends QueryBuilderState {
    dataSpace: DataSpace,
    executionContext: DataSpaceExecutionContext,
    dataSpaceRepo: DataSpacesBuilderRepoistory | undefined,
-    onDataSpaceChange: (val: DataSpaceInfo) => void,
+    onDataSpaceChange: (val: DataSpaceInfo) => Promise<void>,
+    isLightGraphEnabled: boolean,


what is it for as it's not actually used

YannanGao-gs · 2024-09-04T18:11:31Z

packages/legend-application-query/src/stores/QueryEditorStore.ts

@@ -409,6 +411,10 @@ export abstract class QueryEditorStore {
    // do nothing
  }

+  requiresGraphBuilding(): boolean {


currently not used

YannanGao-gs · 2024-09-04T18:40:33Z

packages/legend-graph/src/graph-manager/protocol/pure/v1/V1_PureGraphManager.ts

+      // build types
+      buildState.setMessage(`Building domain models...`);
+      await this.buildTypes(graph, buildInputs, options);
+      stopWatch.record(


necessary elements:
class, function, service,
(path of runtime, connection, store)

await this.buildOtherElements(graph, buildInputs, options); stopWatch.record( GRAPH_MANAGER_EVENT.GRAPH_BUILDER_BUILD_OTHER_ELEMENTS__SUCCESS, );

to resolve dataspace executables

YannanGao-gs · 2024-09-05T19:44:54Z

packages/legend-graph/src/graph-manager/protocol/pure/v1/V1_PureGraphManager.ts

+
+      // build other elements
+      buildState.setMessage(`Building other elements...`);
+      await this.buildOtherElements(graph, buildInputs, options);


it will fail with dataspaces that contain diagrams

YannanGao-gs · 2024-09-06T17:49:04Z

packages/legend-extension-dsl-data-space/src/stores/query-builder/DataSpaceQueryBuilderState.ts

+    queryBuilderState?.dataSpaceAnalysisResult?.executionContextsIndex.get(
+      queryBuilderState.executionContext.name,
+    )?.mappingModelCoverageAnalysisResult;
+  if (


HEADS-UP: This function is not the same as the one in #2523

YannanGao-gs · 2024-09-13T19:20:42Z

...-data-space/src/graph-manager/protocol/pure/v1/V1_DSL_DataSpace_PureGraphManagerExtension.ts

+    const pmcd = mappingModelCoverageAnalysisResult?.model;
+    if (pmcd && projectInfo) {
+      graphEntities = pmcd.elements
+        .concat(mappingModels)


include empty mapping and runtime

YannanGao-gs requested a review from a team as a code owner August 30, 2024 21:14

YannanGao-gs marked this pull request as draft August 30, 2024 21:14

YannanGao-gs self-assigned this Aug 30, 2024

YannanGao-gs force-pushed the performanceEnhance branch 3 times, most recently from 2375853 to b8520b0 Compare September 3, 2024 20:27

YannanGao-gs marked this pull request as ready for review September 3, 2024 20:27

YannanGao-gs mentioned this pull request Sep 3, 2024

Use PMCD returned by mappinganalysis to build minimal graph for query #2523

Open

3 tasks

YannanGao-gs marked this pull request as draft September 3, 2024 20:29

YannanGao-gs force-pushed the performanceEnhance branch 3 times, most recently from aab4beb to e84388e Compare September 4, 2024 13:26

YannanGao-gs commented Sep 4, 2024

View reviewed changes

Use PMCD returned by mapping analysis to build minimal graph for query

0e47202

YannanGao-gs force-pushed the performanceEnhance branch from e84388e to 0e47202 Compare September 5, 2024 19:44

YannanGao-gs commented Sep 5, 2024

View reviewed changes

YannanGao-gs added 2 commits September 5, 2024 17:53

bypass diagrams

f4dfc4a

fix

ca56d82

YannanGao-gs force-pushed the performanceEnhance branch from c4c51c2 to ca56d82 Compare September 6, 2024 13:49

YannanGao-gs added 2 commits September 6, 2024 10:55

enable performance enhancement on template query visitor

7e0fa29

fix class dropdown

136f682

YannanGao-gs commented Sep 6, 2024

View reviewed changes

YannanGao-gs force-pushed the performanceEnhance branch from e9fe026 to 136f682 Compare September 11, 2024 16:19

YannanGao-gs commented Sep 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Use PMCD returned by mappinganalysis to build minimal graph for query #3488

WIP: Use PMCD returned by mappinganalysis to build minimal graph for query #3488

YannanGao-gs commented Aug 30, 2024

changeset-bot bot commented Aug 30, 2024 •

edited

Loading

codecov bot commented Aug 30, 2024 •

edited

Loading

YannanGao-gs Sep 4, 2024 •

edited

Loading

YannanGao-gs commented Sep 4, 2024 •

edited

Loading

YannanGao-gs Sep 4, 2024

YannanGao-gs Sep 4, 2024

YannanGao-gs Sep 4, 2024

YannanGao-gs Sep 4, 2024

YannanGao-gs Sep 4, 2024

YannanGao-gs Sep 4, 2024

YannanGao-gs Sep 4, 2024

YannanGao-gs Sep 5, 2024 •

edited

Loading

YannanGao-gs Sep 6, 2024

YannanGao-gs Sep 13, 2024

WIP: Use PMCD returned by mappinganalysis to build minimal graph for query #3488

Are you sure you want to change the base?

WIP: Use PMCD returned by mappinganalysis to build minimal graph for query #3488

Conversation

YannanGao-gs commented Aug 30, 2024

Summary

How did you test this change?

changeset-bot bot commented Aug 30, 2024 • edited Loading

🦋 Changeset detected

codecov bot commented Aug 30, 2024 • edited Loading

Codecov Report

YannanGao-gs Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

YannanGao-gs commented Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YannanGao-gs Sep 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

changeset-bot bot commented Aug 30, 2024 •

edited

Loading

codecov bot commented Aug 30, 2024 •

edited

Loading

YannanGao-gs Sep 4, 2024 •

edited

Loading

YannanGao-gs commented Sep 4, 2024 •

edited

Loading

YannanGao-gs Sep 5, 2024 •

edited

Loading