[Dataset quality] remove usage of dataStreamStats API in serverless #192839

yngrdyn · 2024-09-13T10:30:30Z

Closes #192166.

This PR aims to remove the usage of DataStreamStats API in serverless. Aditionally we now make use of the new verbose mode for DataStream API which allow us to still show the lastActivity for serverless projects. The change was implemented for both flavours.

Visually it won't represent any changes at the moment, but here are the two flavours still working as before

Screen.Recording.2024-09-13.at.12.38.00.mov

…API for lastActivity

obltmachine · 2024-09-13T10:30:43Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

…ats-api-usage-on-serverless

yngrdyn · 2024-09-17T13:26:43Z

We discovered a bug in the privileges mode regarding verbose mode for dataStreams API. The fix is coming in elastic/elasticsearch#112973

…ats-api-usage-on-serverless

yngrdyn · 2024-09-18T10:43:32Z

.../observability_solution/dataset_quality/public/components/common/insufficient_privileges.tsx

+  const handleButtonClick = useCallback(
+    () => togglePopover(!isPopoverOpen),
+    [isPopoverOpen, togglePopover]
+  );


Caught a bug where user was not able to close the warning explanation on the privileges

Screen.Recording.2024-09-18.at.12.20.34.mov

yngrdyn · 2024-09-18T10:44:10Z

...lution/dataset_quality/public/components/dataset_quality/summary_panel/datasets_activity.tsx

@@ -26,7 +25,7 @@ export function DatasetsActivity() {
      tooltip={summaryPanelDatasetsActivityTooltipText}
      value={text}
      isLoading={isDatasetsActivityLoading}
-      isUserAuthorizedForDataset={isUserAuthorizedForDataset}
+      isUserAuthorizedForDataset={true}


Last Activity is now always enabled through dataStreams API verbose mode

Can we completely remove this prop in favour of this change now ?

This generic component is used also for Estimated Size. In https://github.com/elastic/logs-dev/issues/179 I'll do a refactor

yngrdyn · 2024-09-18T10:45:43Z

.../observability_solution/dataset_quality/server/routes/data_streams/get_data_streams/index.ts

@@ -59,13 +61,15 @@ export async function getDataStreams(options: {
  const mappedDataStreams = filteredDataStreams.map((dataStream) => ({
    name: dataStream.name,
    integration: dataStream._meta?.package?.name,
+    // @ts-expect-error
+    lastActivity: dataStream.maximum_timestamp,


package of elasticsearch has not been released yet, this will be removed when the new package is in place

Can you update the comment like, remove this error post a date or PR merge. This way its trackable in code as this comment will get lost in time

Don't worry, I won't forget. As soon as 8.16 is released I'll get rid of it

…ats-api-usage-on-serverless

.../observability_solution/dataset_quality/public/components/common/insufficient_privileges.tsx

...vability_solution/dataset_quality/server/routes/data_streams/get_data_streams_stats/index.ts

tonyghiani

Left a couple of minor suggestions but overall LGTM!

...ability_solution/dataset_quality/server/routes/data_streams/get_data_stream_details/index.ts

…ge-on-serverless' of github.com:yngrdyn/kibana into 192166-dataset-quality-remove-datastreams-stats-api-usage-on-serverless

…ats-api-usage-on-serverless

achyutjhunjhunwala

LGTM 👍🏼

…ats-api-usage-on-serverless

kibana-ci · 2024-09-19T17:58:23Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 3e37cba
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-192839-3e37cbad346f

Failed CI Steps

Jest Tests #14

Test Failures

[job] [logs] Jest Tests #14 / Configuration button renders the tooltip correctly when hovering the button

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`datasetQuality`	208.7KB	208.5KB	-237.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…lastic#192839) Closes elastic#192166. This PR aims to remove the usage of `DataStreamStats` API in serverless. Aditionally we now make use of the new verbose mode for `DataStream` API which allow us to still show the `lastActivity` for serverless projects. The change was implemented for both flavours. Visually it won't represent any changes at the moment, but here are the two flavours still working as before https://github.com/user-attachments/assets/18cc9b14-24f1-438a-ac07-565c3e728991 (cherry picked from commit 444f95e)

kibanamachine · 2024-09-19T18:52:26Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

remove usage of dataStreamStats API in serverless + using dataStream …

501b092

…API for lastActivity

yngrdyn requested a review from a team as a code owner September 13, 2024 10:30

botelastic bot added the ci:project-deploy-observability Create an Observability project label Sep 13, 2024

yngrdyn added the release_note:skip Skip the PR/issue when compiling release notes label Sep 13, 2024

yngrdyn added the backport:prev-minor Backport to the previous minor version (i.e. one version back from main) label Sep 13, 2024

yngrdyn added 2 commits September 13, 2024 12:32

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

b69c604

…ats-api-usage-on-serverless

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

0ff68b8

…ats-api-usage-on-serverless

yngrdyn added 8 commits September 17, 2024 19:56

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

2481d61

…ats-api-usage-on-serverless

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

803cf2c

…ats-api-usage-on-serverless

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

8772af7

…ats-api-usage-on-serverless

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

2cb6646

…ats-api-usage-on-serverless

ts-errors are expected because elastic client has not being updated yet

3dfbd99

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

a5b76b8

…ats-api-usage-on-serverless

with dataStream API verbose mode, last activity is always available

733a687

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

10f16e3

…ats-api-usage-on-serverless

yngrdyn commented Sep 18, 2024

View reviewed changes

yngrdyn added 4 commits September 18, 2024 13:34

fixing type error

472f9b9

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

398f95d

…ats-api-usage-on-serverless

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

38bb9bb

…ats-api-usage-on-serverless

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

9a104c6

…ats-api-usage-on-serverless

tonyghiani self-requested a review September 19, 2024 08:18

tonyghiani reviewed Sep 19, 2024

View reviewed changes

.../observability_solution/dataset_quality/public/components/common/insufficient_privileges.tsx Outdated Show resolved Hide resolved

tonyghiani reviewed Sep 19, 2024

View reviewed changes

...vability_solution/dataset_quality/server/routes/data_streams/get_data_streams_stats/index.ts Outdated Show resolved Hide resolved

tonyghiani reviewed Sep 19, 2024

View reviewed changes

...vability_solution/dataset_quality/server/routes/data_streams/get_data_streams_stats/index.ts Show resolved Hide resolved

tonyghiani approved these changes Sep 19, 2024

View reviewed changes

achyutjhunjhunwala reviewed Sep 19, 2024

View reviewed changes

...ability_solution/dataset_quality/server/routes/data_streams/get_data_stream_details/index.ts Show resolved Hide resolved

achyutjhunjhunwala reviewed Sep 19, 2024

View reviewed changes

...ability_solution/dataset_quality/server/routes/data_streams/get_data_stream_details/index.ts Show resolved Hide resolved

yngrdyn added 6 commits September 19, 2024 11:39

PR review comments

cd56be8

Merge branch '192166-dataset-quality-remove-datastreams-stats-api-usa…

7a0cd44

…ge-on-serverless' of github.com:yngrdyn/kibana into 192166-dataset-quality-remove-datastreams-stats-api-usage-on-serverless

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

f161b78

…ats-api-usage-on-serverless

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

1d00819

…ats-api-usage-on-serverless

fixing build after refactor

b2a6926

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

8216546

…ats-api-usage-on-serverless

achyutjhunjhunwala approved these changes Sep 19, 2024

View reviewed changes

yngrdyn added 2 commits September 19, 2024 16:06

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

94a53ac

…ats-api-usage-on-serverless

Merge branch 'main' into 192166-dataset-quality-remove-datastreams-st…

3e37cba

…ats-api-usage-on-serverless

yngrdyn merged commit 444f95e into elastic:main Sep 19, 2024
23 of 24 checks passed

kibanamachine added the v9.0.0 label Sep 19, 2024

kibanamachine mentioned this pull request Sep 19, 2024

[8.x] [Dataset quality] remove usage of dataStreamStats API in serverless (#192839) #193468

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dataset quality] remove usage of dataStreamStats API in serverless #192839

[Dataset quality] remove usage of dataStreamStats API in serverless #192839

yngrdyn commented Sep 13, 2024 •

edited by kibanamachine

Loading

obltmachine commented Sep 13, 2024

yngrdyn commented Sep 17, 2024

yngrdyn Sep 18, 2024

yngrdyn Sep 18, 2024

achyutjhunjhunwala Sep 19, 2024

yngrdyn Sep 19, 2024

yngrdyn Sep 18, 2024

achyutjhunjhunwala Sep 19, 2024

yngrdyn Sep 19, 2024

tonyghiani left a comment

achyutjhunjhunwala left a comment

kibana-ci commented Sep 19, 2024 •

edited

Loading

kibanamachine commented Sep 19, 2024

[Dataset quality] remove usage of dataStreamStats API in serverless #192839

[Dataset quality] remove usage of dataStreamStats API in serverless #192839

Conversation

yngrdyn commented Sep 13, 2024 • edited by kibanamachine Loading

obltmachine commented Sep 13, 2024

🤖 GitHub comments

yngrdyn commented Sep 17, 2024

yngrdyn Sep 18, 2024

Choose a reason for hiding this comment

yngrdyn Sep 18, 2024

Choose a reason for hiding this comment

achyutjhunjhunwala Sep 19, 2024

Choose a reason for hiding this comment

yngrdyn Sep 19, 2024

Choose a reason for hiding this comment

yngrdyn Sep 18, 2024

Choose a reason for hiding this comment

achyutjhunjhunwala Sep 19, 2024

Choose a reason for hiding this comment

yngrdyn Sep 19, 2024

Choose a reason for hiding this comment

tonyghiani left a comment

Choose a reason for hiding this comment

achyutjhunjhunwala left a comment

Choose a reason for hiding this comment

kibana-ci commented Sep 19, 2024 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

History

kibanamachine commented Sep 19, 2024

💚 All backports created successfully

Questions ?

yngrdyn commented Sep 13, 2024 •

edited by kibanamachine

Loading

kibana-ci commented Sep 19, 2024 •

edited

Loading