Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add image support to LiteLLMEmbedder #433

Closed
ludwiktrammer opened this issue Mar 20, 2025 · 1 comment
Closed

feat: add image support to LiteLLMEmbedder #433

ludwiktrammer opened this issue Mar 20, 2025 · 1 comment
Assignees
Labels
feature New feature or request
Milestone

Comments

@ludwiktrammer
Copy link
Collaborator

ludwiktrammer commented Mar 20, 2025

LiteLLMEmbedder should support image embeddings when used with embedding models that support image embeddings.

We previously attempted to add this capability, but LiteLLM had issues that made the multimodal embedder API completely inconsistent with the API of other models. This made it exceedingly difficult to integrate support for these models into our existing LiteLLM client class. As a temporary workaround, we introduced VertexAIMultimodelEmbedder, which also uses LiteLLM behind the scenes but only supports a single family of multimodal models.

Given the current state of LiteLLM, it appears that the upstream issues have been resolved, making it relatively straightforward to add image support to our LiteLLMEmbedder class.

Definition of Done

  1. LiteLLMEmbedder should support embedding images via the embed_image method when configured with a model that supports it.
  2. The image_support method in LiteLLMEmbedder should correctly indicate whether the model supports image embeddings. (Note: In cases of uncertainty, false negatives are preferable to false positives.)
  3. The VertexAIMultimodelEmbedder class should be removed, and all instances of its use should be replaced by LiteLLMEmbedder configured with the vertex_ai/multimodalembedding model.
@ludwiktrammer ludwiktrammer added the feature New feature or request label Mar 20, 2025
@ludwiktrammer ludwiktrammer self-assigned this Mar 20, 2025
@ludwiktrammer ludwiktrammer moved this to Backlog in ragbits Mar 20, 2025
@ludwiktrammer ludwiktrammer moved this from Backlog to Ready in ragbits Mar 24, 2025
@mhordynski mhordynski added this to the Ragbits 1.0.0 milestone Mar 25, 2025
@kzamlynska kzamlynska moved this from Ready to In Progress in ragbits Mar 26, 2025
@kzamlynska
Copy link
Collaborator

There is still a problem with liteLLM library:

  • we can't distinguish between models supporting images and those one which not.
  • some of the models are able to work with list of objects for embedding some not - liteLLM doesn't deal with this problem.
    The solution which we have know with separate vertex_multimodal class is good at this moment and the possible improvements will make the code more complicated.
    Decision: we won't work on this problem now.

@github-project-automation github-project-automation bot moved this from In Progress to Done in ragbits Apr 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants