Features/distances #694
Features/distances #694
Conversation
There was a problem hiding this comment.
Hey @VascoSch92
Thank you so much for the contribution! This is looking really good.
I think the logic for the distance calculation is good. We need to make this transformer look like the other transformers that we have in the library.
Why don't you have a look at the class RelativeFeatures in the creation module, and try to incorporate that logic in this class as well? Mostly related to which parameters we need in the init, and which checks we normally do. And there you will see as well how you can import many premade bits of text for the documentation.
Let me know how you get along! Thank you!
| _check_param_drop_original(drop_original=drop_original) | ||
| self.drop_original = drop_original | ||
|
|
||
| self.variables = None |
There was a problem hiding this comment.
I don't think we need this parameter. I'd suggest using RelativeFeatures as template to model this class: https://github.com/VascoSch92/feature_engine/blob/e1e927625678ee73c5c3a9edcf79e955ff9c5e8e/feature_engine/creation/relative_features.py
There was a problem hiding this comment.
variables is a parameter that we have in all transformers, so I would stick to this name instead of using coordinate_columns
In short, let's replace coordinate_columns by variables.
|
Is it possible to calculate Haversine distance using sklearn? It is quite fast and well optimized, reimplementing it seems like a not so good idea. P.S. it could be quite interesting to add more measures of distance, for example Ruler distance |
Yes it is possible to compute the Haversine distance with sklearn. I was also thinking to use an apply and the Haversine distance method of Sklearn. The question is: is it faster than vectorisation? But I'm happy to change if it faster or If there is a faster method than mine ;-) |
I'm not sure I understand the question. |
I think the issue with the sklearn implementation is that it does a cartesian product between X and Y and yields a matrix. We only need a pairwise calculation between X and Y that yields a vector. |
|
|
I know that it is a simple way to code this, but from a time complexity perspective, it's not a great idea to use quadratic complexity when only linear complexity is needed. |
Yea, you are right, this way it will be better |
Hey @glevv thanks for the suggestion. If I understood this blog correctly, it has 3 computations: euclidean, harvesine (the one we are trying to implement here) and a more complicated one that has a smaller error (vincenty's formula). Is this correct? I'd suggest we stick to harvesine in this PR, and see if we create an issue to expand the class later with the Vincenty's. Is this formula commonly used? do we really need an error as small as 0.5mm for geo coordinates? |
They are all measures of distance between two points on ellipsoid. There were no Vincenty formula, but it's quite heavy to compute. In this particular blog post they talked about two simpler and faster formulas (Cheap Ruler and FCC equation) but with higher error.
Ye, let's go with haversine only, not sure about Vincenty tho |
|
Hey @solegalli |
|
No Problem at all @VascoSch92 . Same here. I am doing some big changes to the correlation transformers, I think we could release a new version when i got those finished, hopefully during February. It would be great if we can squeeze this transformer in that release 2. If you find the time, we look forward to your contribution :) |
Hey @solegalli :-) |
|
Sure! Contributions are welcome any time :) |
|
ok perfect. I will work on it. |
|
Hey @solegalli I still need some guidance for some point:
|
There was a problem hiding this comment.
Hey @VascoSch92
This is looking good. We need to tidy the code a bit. In the init, we can only test that the user enters allowed inputs, and then assign them. We don't need extra functions/methods for this checks.
In the fit, we need to unpack the variables and check the allowed values.
If you could re-arrange that, I can then take another look.
It would be great if you could rebase main after you sync it to the latest version of the repo, because we made a lot of changes since this PR.
Thank you!
| return parameter | ||
|
|
||
| def _check_coordinate_columns( | ||
| self, columns: List[List[Union[str, int]]] |
There was a problem hiding this comment.
Is this list of lists? I think it is just a list.
Just a first sketch.
Let me know what do you think :-)