The world's first barcode purchase dates back to June 26, 1974 - it was a package of chewing gum in one of the US supermarkets. By reading information from a barcode, according to various estimates, it is possible to speed up operations with goods by an average of 30%. Now barcodes are scanned by both sellers, and warehouse workers, and buyers - for example, if they want to make a purchase at a self-service checkout.

In the article, we will consider some features of barcode recognition using the ML Kit library . The material can be useful for both novice developers with basic skills and experienced professionals who want to learn a new tool.

ML Kit is a free mobile SDK from Google that allows you to use machine learning on Android, iOS, and Flutter devices. In mobile development, this is perhaps the easiest way to add neural networks to an application. In turn, this allows you to simplify the implementation of some functions.

Key features of ML Kit:

• Text recognition (including handwritten)

• Translation of text between languages ​​(offline)

• Face (and emotion) recognition

• Object recognition

Also lesser known:

• Pose recognition (determines the location of the head)

• Barcode scanning

Such functions can be useful in many applications, such as tourist guides, for translating signs and signs and displaying information about places of interest. As an example, we once participated in the creation of an application in which tourists could take pictures and recognize data so as not to enter them manually.

So, let's move on to the practice of working with ML Kit. In one of the projects, our partner had a need to replace the barcode scanning library. Previously, the customer used the paid Scandit library and faced some limitations. At that time, in particular, it was required to display the library logo on the screen for scanning codes. Also, the license agreement did not exclude the possibility that the manufacturer could revoke the license. As an alternative solution, the development team chose ML Kit Barcode scanning.

Barcode scanning example (Android)

First of all, before starting work with ML Kit, you need to include the necessary libraries in gradle:

implementation 'com.google.mlkit:barcode-scanning:17.0.0'

Or:

implementation 'com.google.android.gms:play-services-mlkit-barcode-scanning:16.2.1'

And also in manifest:

<meta-data
    android:name="com.google.mlkit.vision.DEPENDENCIES"
	android:value="barcode" />

In the first case, everything you need is added to the installation file, and in the second case, it is dynamically downloaded. Also in the first case, you can count on a little more performance.

Next, you need to prepare the Detector. This is the main interface in ML Kit and has the most important process and close methods: 

• process does all the image processing and returns results that depend on the particular implementation of the interface; 

• with the help of close, we free up the occupied resources.

Consider the process of preparing BarcodeScanner, one of the successors of Detector:

val detector = BarcodeScanning.getClient(
    BarcodeScannerOptions.Builder()
    	.setBarcodeFormats(Barcode.FORMAT_QR_CODE)
    	.build()
)

BarcodeScanning is a variation of the generating pattern. The getClient method accepts the parameters of the required object as input, and produces an instance of BarcodeScanner at the output. In turn, BarcodeScannerOptions is created through the standard Builder. In this case, we have indicated that we want to recognize only QR codes. This approach applies to the rest of the ML Kit features as well.

After that, you can use this Detector, below is a simple example:

detector.process(image).addOnSuccessListener { barcodes ->
    barcodes.firstOrNull()?.let {
    	Toast.makeText(context, it.rawValue, Toast.LENGTH_SHORT).show()
    }
}

Possible difficulties

1) Realtime

While the ML Kit is fairly easy to use, we have found some pitfalls. The main questions turned out to be related to work in realtime mode. At the time of the project, we did not have official examples, so we studied unofficial examples, and some of them had errors. 

So, initially we considered the following example (we will consider the features of obtaining data from the camera below).

private var imageAnalysis = ImageAnalysis.Analyzer { imageProxy ->
    val image = imageProxy.image ?: return@Analyzer
	val inputImage = InputImage.fromMediaImage(imageProxy.image!!, imageProxy.imageInfo.rotationDegrees)
    detector?.process(inputImage)?.addOnSuccessListener { barcodes ->
    	barcodes.firstOrNull()?.let {
        	Toast.makeText(context, it.rawValue, Toast.LENGTH_SHORT).show()
    	}
	}
	imageProxy.close()
}

Although this method worked on most devices, on less powerful devices it led to memory overflow. Since the detector does not process frames instantly, there was a risk of a serious memory leak. For example, if each photo “weighs” 2 megabytes, and there are several hundred frames in memory at the same time, this will crash the application. 

After reviewing the ImageAnalysis documentation, we found that one of the reasons for calling imageProxy.close() is to tell the system that the application is ready to process the next frame.

As a result, we have changed the code as follows:

private var imageAnalysis = ImageAnalysis.Analyzer { imageProxy ->
    val image = imageProxy.image ?: return@Analyzer
	val inputImage = InputImage.fromMediaImage(imageProxy.image!!, imageProxy.imageInfo.rotationDegrees)
    detector?.process(inputImage)?.addOnSuccessListener { barcodes ->
    	barcodes.firstOrNull()?.let {
        	Toast.makeText(context, it.rawValue, Toast.LENGTH_SHORT).show()
    	}
	}?.addOnCompleteListener {
    	imageProxy.close()
	}
}

With this implementation, only one frame was always in memory, and the crash problem on low-performance devices was solved.

2) Adaptation

Also, one of our tasks was to adapt the ML Kit to the needs of a particular project. In particular, the previous library was able to process both black and white and white and black barcodes. In turn, ML Kit did not understand negatives at the start of work.

To solve the problem, we changed the code. Previous option:

InputImage.fromMediaImage(imageProxy.image!!, imageProxy.imageInfo.rotationDegrees)

The new version has become more complex, with pre-processing:

private fun getByteDataFromImage(image: Image): ByteArray? {
    var data: ByteArray? = null
    val planes: Array<Image.Plane> = image.planes
	if (planes.isNotEmpty()) {
    	val buffer: ByteBuffer = planes[0].buffer
    	data = ByteArray(buffer.remaining())
    	buffer.get(data)
    }
    return data
}
getByteDataFromImage(image)?.let { byteArray ->
    processImage(byteArray, imageProxy)
    val inverted = ByteArray(byteArray.size)
    byteArray.forEachIndexed { index, byte ->
    	inverted[index] = byte.inv()
    }
	processImage(inverted, imageProxy)
}

Here we received a picture as an array of bytes and divided it into positive and negative, which were sent separately to the detector.

3)CameraX

Another bug we encountered was related to the incorrect use of resolution in CameraX. We set the maximum resolution to 1920x1080.

val camProfile = CamcorderProfile.get(CamcorderProfile.QUALITY_1080P)
analysisBuilder.setTargetResolution(Size(camProfile.videoFrameWidth,
                                     	camProfile.videoFrameHeight))

However, in CameraX, the default output was 320x640. We found out that the order of width and height depends on the orientation, and for the “portrait” output in our case, we needed the following:

val camProfile = CamcorderProfile.get(CamcorderProfile.QUALITY_1080P)
analysisBuilder.setTargetResolution(Size(camProfile.videoFrameHeight,
                                     	camProfile.videoFrameWidth))

Conclusion

After customization and implementation in the project, we made sure that the ML Kit meets the needs of the application and can replace the previous library. In some cases, the paid library was more effective, for example, on a small proportion of "blurry" barcodes. In turn, the ML Kit library is completely free and does not require you to add your logo to the scan screen. As a result, after testing, the application owner decided to completely switch to ML Kit.