You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
LiteLLMEmbedder should support image embeddings when used with embedding models that support image embeddings.
We previously attempted to add this capability, but LiteLLM had issues that made the multimodal embedder API completely inconsistent with the API of other models. This made it exceedingly difficult to integrate support for these models into our existing LiteLLM client class. As a temporary workaround, we introduced VertexAIMultimodelEmbedder, which also uses LiteLLM behind the scenes but only supports a single family of multimodal models.
Given the current state of LiteLLM, it appears that the upstream issues have been resolved, making it relatively straightforward to add image support to our LiteLLMEmbedder class.
Definition of Done
LiteLLMEmbedder should support embedding images via the embed_image method when configured with a model that supports it.
The image_support method in LiteLLMEmbedder should correctly indicate whether the model supports image embeddings. (Note: In cases of uncertainty, false negatives are preferable to false positives.)
The VertexAIMultimodelEmbedder class should be removed, and all instances of its use should be replaced by LiteLLMEmbedder configured with the vertex_ai/multimodalembedding model.
The text was updated successfully, but these errors were encountered:
we can't distinguish between models supporting images and those one which not.
some of the models are able to work with list of objects for embedding some not - liteLLM doesn't deal with this problem.
The solution which we have know with separate vertex_multimodal class is good at this moment and the possible improvements will make the code more complicated.
Decision: we won't work on this problem now.
LiteLLMEmbedder
should support image embeddings when used with embedding models that support image embeddings.We previously attempted to add this capability, but LiteLLM had issues that made the multimodal embedder API completely inconsistent with the API of other models. This made it exceedingly difficult to integrate support for these models into our existing LiteLLM client class. As a temporary workaround, we introduced
VertexAIMultimodelEmbedder
, which also uses LiteLLM behind the scenes but only supports a single family of multimodal models.Given the current state of LiteLLM, it appears that the upstream issues have been resolved, making it relatively straightforward to add image support to our
LiteLLMEmbedder
class.Definition of Done
LiteLLMEmbedder
should support embedding images via theembed_image
method when configured with a model that supports it.image_support
method inLiteLLMEmbedder
should correctly indicate whether the model supports image embeddings. (Note: In cases of uncertainty, false negatives are preferable to false positives.)VertexAIMultimodelEmbedder
class should be removed, and all instances of its use should be replaced byLiteLLMEmbedder
configured with thevertex_ai/multimodalembedding
model.The text was updated successfully, but these errors were encountered: