Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Apibench] How to evaluate a model by openai API? #592

Open
djstrong opened this issue Aug 21, 2024 · 3 comments
Open

[Apibench] How to evaluate a model by openai API? #592

djstrong opened this issue Aug 21, 2024 · 3 comments
Labels
BFCL-General General BFCL Issue

Comments

@djstrong
Copy link

I have remotely hosted vllm models. How to evaluate them?
@djstrong djstrong added the apibench-data APIBench data label Aug 21, 2024
@HuanzhiMao HuanzhiMao added BFCL-General General BFCL Issue and removed apibench-data APIBench data labels Aug 22, 2024
@HuanzhiMao
Copy link
Collaborator

Hey @djstrong, just want to double-check, are you referring to the evaluation in Apibench or the Berkeley Function Calling Leaderboard (BFCL)?

@djstrong
Copy link
Author

Sorry, I mean BFCL.

@HuanzhiMao
Copy link
Collaborator

Take a look at the instructions here. Let me know if you have more questions!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
BFCL-General General BFCL Issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@djstrong @HuanzhiMao