If I use moganet to train a model, and then use the model for feature extraction and image retrieval, how should I apply this framework?

[goldwater668]

Is there an existing issue for this?

• I have searched the existing issues.

Is your feature request related to a problem? Please describe.

If I use moganet to train a model, and then use the model for feature extraction and image retrieval, how should I apply this framework?

Describe the solution you'd like.

If I use moganet to train a model, and then use the model for feature extraction and image retrieval, how should I apply this framework?

Describe an alternate solution.

If I use moganet to train a model, and then use the model for feature extraction and image retrieval, how should I apply this framework?

Anything else? (Additional Context)

If I use moganet to train a model, and then use the model for feature extraction and image retrieval, how should I apply this framework?

[junjiejiangjjj]

hi @goldwater668 , towhee integrates image models through timm. If your model can be called through timm, use the following code:

from towhee import pipe, ops, DataCollection

p = (
    pipe.input('path')
        .map('path', 'img', ops.image_decode())
        .map('img', 'vec', ops.image_embedding.timm(model_name='model_name', checkpoint_path="your local weights"))
        .output('img', 'vec')
)

DataCollection(p('towhee.jpeg')).show()

If not, you need to implement an operator yourself, https://towhee.readthedocs.io/en/latest/operator/usage.html#custom-operators

[goldwater668]

@junjiejiangjjj
https://github.com/Westlake-AI/MogaNet
This is the warehouse of the model. I saw the following call. It should support timm. If you have time, you can help and see if it can be called directly.

from timm.models import create_model, apply_test_time_pool, load_checkpoint, is_model, list_models
from timm.data import create_dataset, create_loader, resolve_data_config, RealLabelsImagenet
from timm.utils import accuracy, AverageMeter, natural_key, setup_default_logging, set_jit_legacy
checkpoint = torch.load(checkpoint_path)
model = create_model(
        "moganet_xtiny",
        pretrained=False,
        num_classes=2,
        in_chans=3,
        global_pool=None,
        scriptable=False,
        mem_index=[10],
        K=1024)
model.load_state_dict(checkpoint["state_dict"], strict=True)
model = model.cuda()
model.eval()

transform = create_transform(input_size=(3, 224, 224),mean=(0.485, 0.456, 0.406),std=(0.229, 0.224, 0.225),crop_pct=0.9)
for root, subdirs, files in os.walk(folder, topdown=False, followlinks=True):
    rel_path = os.path.relpath(root, folder) if (root != folder) else ''
    label = os.path.basename(rel_path) if True else rel_path.replace(os.path.sep, '_')
    for f in files:
        base, ext = os.path.splitext(f)
        if ext.lower() in ['.png', '.jpg', '.jpeg']:
            image_path = os.path.join(root,f)
            img=Image.open(image_path)
            img = img.convert("RGB")
            input_image = transform(img).unsqueeze(0)
            input_image = Variable(input_image).to(DEVICE)
            out = model(input_image)
            img_num+=1
            classes = ('NG', 'OK')
            _, pred = torch.max(out.data, 1)

[junjiejiangjjj]

It seems that you need to run it in the directory of this project.

git clone https://github.com/Westlake-AI/MogaNet 
cd MogaNet 

import models
from towhee import pipe, ops, DataCollection

p = (
    pipe.input('path')
        .map('path', 'img', ops.image_decode())
        .map('img', 'vec', ops.image_embedding.timm(model_name='moganet_xtiny', checkpoint_path="your local weights"))
        .output('img', 'vec')
)

DataCollection(p('towhee.jpeg')).show()

[junjiejiangjjj]

hi @goldwater668 , if you change the channels, you also need to change it when initializing the model.

[junjiejiangjjj]

This is determined by the model. This model has these specifications

[junjiejiangjjj]

Towhee accelerates model inference through Triton Server , but only supports a subset of models. For your own models, you can consider converting them to ONNX to accelerate inference.

[goldwater668]

@junjiejiangjjj
Where is the similarity threshold for retrieval set?Then how much memory does the inserted data occupy and where can I check it?