Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Select which adblock data files to generate using cli args #699

Closed

Conversation

tontinton
Copy link

No description provided.

@tontinton tontinton changed the title Faster generate adblock data Generate adblock data faster Sep 20, 2023
@tontinton tontinton changed the title Generate adblock data faster Generate adblock data concurrently Sep 20, 2023
Copy link
Collaborator

@antonok-edm antonok-edm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tontinton thanks for submitting this!

Unfortunately there are a few dependencies between components that are left in for historical reasons, e.g. the regional data files depend on the default component files here.

I'm hoping to get rid of those dependencies eventually, but I don't want to suddenly break things for older browser versions that haven't updated in a while. In the meantime we still have to do most of these in order.

So, I can't merge this just yet, but hopefully I can get to it in the future.

† I say "most" because the test data files can definitely be generated in parallel... but those can probably be removed altogether, too. I don't believe we have any code that would be using them anymore (#701).

@tontinton tontinton force-pushed the faster-generate-adblock-data branch 2 times, most recently from 2f76717 to 04d89fc Compare October 5, 2023 11:21
@tontinton tontinton force-pushed the faster-generate-adblock-data branch 3 times, most recently from 549bb86 to 0c772df Compare October 5, 2023 11:29
@tontinton
Copy link
Author

Ok I updated the PR to only include the cli stuff, will be useful for me (I want to generate only the default list)

@tontinton tontinton changed the title Generate adblock data concurrently Select which adblock data files to generate using cli args Oct 5, 2023
@tontinton tontinton force-pushed the faster-generate-adblock-data branch 3 times, most recently from 8d9f2a9 to 7c9a022 Compare October 12, 2023 14:25
@tontinton
Copy link
Author

tontinton commented Dec 20, 2023

bumping, still keeping a fork :(

regional lists sometimes return 404, while the default seems to work all the time.

@tontinton
Copy link
Author

another final bump? 😅

@antonok-edm
Copy link
Collaborator

@tontinton sorry for the delay, but the reorganization in #844 should make this significantly easier. All the components will be "self-contained", i.e. not dependent on other component jobs running.

Going forward:

  • once merged you could immediately patch getAdblockComponentDescriptors to skip the components you're not interested in
  • later, I'm planning to move the per-component classes to exports in their own files; making the packaging scripts themselves basically just these lines. Then you can treat this repo as a library dependency and define a custom entrypoint to drive the logic of which components to package.

@tontinton tontinton closed this Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants