The Vision Transformer (ViT) is rapidly replacing convolutional neural networks due to its simplicity, flexibility, and scalability. An image is segmented into patches, and each patch is linearly projected onto a token, forming the basis of this model. Input photos are usually squared and divided into a set number of patches before being used.
Recent publications have investigated potential deviations from this model: FlexiViT allows for a continuous range of sequence lengths and therefore calculates the cost by accommodating different patch sizes in a single design. This is accomplished by randomly selecting a patch size during each training iteration and using a scaling technique to accommodate many patch sizes in the initial convolutional embedding. Pix2Struct’s alternative correction approach, which maintains the aspect ratio, is invaluable for tasks such as understanding graphics and documents.
NaViT is an alternative developed by Google researchers. Patch n’ Pack is a technique to vary the resolution while maintaining the aspect ratio by grouping many patches of separate images into a single sequence. This idea is based on “example bundling”, a technique used in natural language processing to efficiently train models with variable-length inputs by combining multiple instances into a single sequence. Scientists have found evidence of;
A significant amount can reduce training time by randomly sampling resolutions. NaViT achieves excellent performance across a wide range of solutions, facilitating a smooth cost-performance trade-off at inference time, and is easily adaptable at low cost to new work.
Research ideas such as aspect ratio-preserving resolution sampling, variable token drop rates, and adaptive computing emerge from the fixed batch shapes made possible by example packing.
NaViT’s computational efficiency is particularly impressive during pre-training and persists through adjustments. The successful application of a single NaViT on different resolutions allows a smooth trade-off between performance and inference cost.
Feeding data into a deep neural network during training and batch operation is common practice. Therefore, computer vision applications should use predetermined batch sizes and geometries to ensure optimal performance on existing hardware. Due to this and the inherent architectural constraints of convolutional neural networks, it has become common to scale or fill images to a predetermined size.
Although NaViT is based on the original ViT, any ViT variant capable of processing a sequence of patches can be used in theory. Researchers implement the following structural changes to support Patch n’ Pack. Patch n’ Pack is a simple visual transformer sequence conditioning app that dramatically increases training effectiveness, as proven by the research community. The resulting NaViT models are flexible and easy to adapt to new jobs without breaking the bank. Adaptive computing research and new algorithms to improve training and inference efficiency are just two examples of the research made possible by Patch n’ Pack, which was previously hampered by the need for fixed batch forms. . They also see NaViT as a step in the right direction for ViTs, as it represents a change from the conventional input and modeling pipeline of most computer vision models, designed by CNN.
Check Paper. All credit for this research goes to the researchers of this project. Also don’t forget to register. our 26k+ ML subreddit, Discord ChannelAnd E-mailwhere we share the latest AI research news, cool AI projects, and more.
🚀 Discover more than 800 AI tools in AI Tools Club
Dhanshree Shenwai is a Computer Engineer with good experience in FinTech companies spanning Finance, Cards & Payments and Banking with keen interest in AI applications. She is enthusiastic about exploring new technologies and advancements in today’s changing world, making everyone’s life easier.
