Rescan for an augmented object

[sach1n1]

Hallo Community,

I was able to get the augmented images example from the repository working with a custom image and custom 3D object. In my current application the image when augmented, it has the 3D object placed upon it which disappears after 5 seconds. Once the object has disappeared, I want to rescan the image and augment it again. By default in the sample project, this had been disabled by changing the value of a boolean flag rabbitDetected. This boolean flag is set to true once the image has been detected and prevents it from detecting the image again. However, when I change this value to false in my code and try to augment it, the app becomes extremely laggy (evetually unresponsive) and the augmented object does not disappear after five seconds. I was not able to understand this phenomenon. I will attach my code for reference.

Thank You. :)

import android.graphics.BitmapFactory
import android.net.Uri
import android.os.Bundle
import android.os.Handler
import android.os.Looper
import android.view.View
import android.view.ViewGroup.MarginLayoutParams
import android.widget.Toast
import androidx.appcompat.app.AppCompatActivity
import androidx.appcompat.widget.Toolbar
import androidx.core.view.ViewCompat
import androidx.core.view.WindowInsetsCompat
import androidx.fragment.app.Fragment
import androidx.fragment.app.FragmentManager
import androidx.fragment.app.FragmentOnAttachListener
import com.google.ar.core.*
import com.google.ar.sceneform.AnchorNode
import com.google.ar.sceneform.Sceneform
import com.google.ar.sceneform.math.Vector3
import com.google.ar.sceneform.rendering.ModelRenderable
import com.google.ar.sceneform.ux.ArFragment
import com.google.ar.sceneform.ux.BaseArFragment.OnSessionConfigurationListener
import com.google.ar.sceneform.ux.InstructionsController
import com.google.ar.sceneform.ux.TransformableNode
import java.util.*
import java.util.concurrent.CompletableFuture
import java.util.function.Consumer

class MainActivity : AppCompatActivity(), FragmentOnAttachListener, OnSessionConfigurationListener {
    private val futures: MutableList<CompletableFuture<Void>> = ArrayList()
    private var arFragment: ArFragment? = null
    private var robotDetected = false
    private var database: AugmentedImageDatabase? = null
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        val toolbar = findViewById<Toolbar>(R.id.toolbar)
        setSupportActionBar(toolbar)

        ViewCompat.setOnApplyWindowInsetsListener(toolbar) { _: View?, insets: WindowInsetsCompat ->
            (toolbar.layoutParams as MarginLayoutParams).topMargin = insets
                .getInsets(WindowInsetsCompat.Type.systemBars()).top
            WindowInsetsCompat.CONSUMED
        }

        supportFragmentManager.addFragmentOnAttachListener(this)
        if (savedInstanceState == null) {
            if (Sceneform.isSupported(this)) {
                supportFragmentManager.beginTransaction()
                    .add(R.id.arFragment, ArFragment::class.java, null)
                    .commit()
            }
        }
    }

    override fun onAttachFragment(fragmentManager: FragmentManager, fragment: Fragment) {
        if (fragment.id == R.id.arFragment) {
            arFragment = fragment as ArFragment
            arFragment!!.setOnSessionConfigurationListener(this)
        }
    }

    override fun onSessionConfiguration(session: Session, config: Config) {
        // Disable plane detection
        config.planeFindingMode = Config.PlaneFindingMode.DISABLED

        // Images to be added in AugmentedImageDatabase
        database = AugmentedImageDatabase(session)
        val robotImage = BitmapFactory.decodeResource(resources, R.drawable.robot)
        database!!.addImage("robot", robotImage)
        config.augmentedImageDatabase = database

        // Check for image detection
        arFragment!!.setOnAugmentedImageUpdateListener { augmentedImage: AugmentedImage ->
            onAugmentedImageTrackingUpdate(
                augmentedImage
            )
        }
    }

    override fun onDestroy() {
        super.onDestroy()
        futures.forEach(Consumer { future: CompletableFuture<Void> ->
            if (!future.isDone) future.cancel(
                true
            )
        })
    }

    private fun onAugmentedImageTrackingUpdate(augmentedImage: AugmentedImage) {
        //If robot image already detected, we can set it to false so the tracking takes place again. 
        if (robotDetected) {
            robotDetected = false  //this line is not there in the actual sample
            // and this line makes the app laggy.
            return
        }
        
        if (augmentedImage.trackingState == TrackingState.TRACKING
        ) {
            // Setting anchor to the center of Augmented Image
            val anchorNode1 = AnchorNode(augmentedImage.createAnchor(augmentedImage.centerPose))
            if (!robotDetected && augmentedImage.name == "robot") {
                robotDetected = true
                Toast.makeText(this, "Robot tag detected", Toast.LENGTH_SHORT).show()
                anchorNode1.worldScale = Vector3(0.2f, 0.2f, 0.2f)
                arFragment!!.arSceneView.scene.addChild(anchorNode1)
                //Toast.makeText(this, "UbiLab", Toast.LENGTH_SHORT).show()
                futures.add(ModelRenderable.builder()
                    .setSource(this, Uri.parse("models/text.glb"))
                    .setIsFilamentGltf(true)
                    .build()
                    .thenAccept { textModel: ModelRenderable? ->
                        val modelNode = TransformableNode(
                            arFragment!!.transformationSystem
                        )
                        modelNode.renderable = textModel
                        anchorNode1.addChild(modelNode)
                    }
                    .exceptionally { throwable: Throwable? ->
                        Toast.makeText(
                            this,
                            "Unable to load rabbit model",
                            Toast.LENGTH_LONG
                        )
                            .show()
                        null
                    })

                Handler(Looper.getMainLooper()).postDelayed({
                    arFragment!!.arSceneView.scene.removeChild(anchorNode1)
                }, 5000)
            }
        }

        if (robotDetected) {
            arFragment!!.instructionsController.setEnabled(
                InstructionsController.TYPE_AUGMENTED_IMAGE_SCAN, true
            )
        }
    }
}

[fvito]

Hi,
I suspect the model disappears after 5s because of this piece of code
Handler(Looper.getMainLooper()).postDelayed({ arFragment!!.arSceneView.scene.removeChild(anchorNode1) }, 5000)
This makes is so, after 5s, the node with the model is removed from the scene. And because robotDetected flag is not reset, new model is not added.

As for the second question. If you make it so that robotDetected , the code will add a new model to the scene every frame (or every time it recognizes the image which might be every frame).. This can result in a lot of models, thus making the app lag.

[sach1n1]

Hallo @fvito ,

I want the model to disappear five seconds after image has been augmented.

As for the second question. If you make it so that robotDetected , the code will add a new model to the scene every frame (or every time it recognizes the image which might be every frame).. This can result in a lot of models, thus making the app lag.

Would it be possible to scan the same image again, once it has been already detected by the application in the same session?

[RGregat]

It should work if you set robotDetected back to false

Handler(Looper.getMainLooper()).postDelayed({
   arFragment!!.arSceneView.scene.removeChild(anchorNode1)
   robotDetected = false           
}, 5000)

[sach1n1]

Hallo @RGregat ,

I tried out your suggestion, it did not solve the problem, however, I have some quite interesting observations:

1. I initially thought it wouldn't work as I expected it to cause the app to slow down and lag (as was the case when I set it to false initially), however I was completely surprised when in fact they app continued to work without lag. But the object remained in the position and just refreshed after every five seconds, even without me scanning the image again. This meant that once an image has been detected, it is always kind of in an ON state. This can be seen in the first attached video where the toast message shows "Robot Tag Detected". Also I have a doubt, how does it differ from my implementation where I was setting the robotDetected to false in the function:
   if (robotDetected) { robotDetected = false //this line is not there in the actual sample // and this line makes the app laggy. return }

1.mp4

2. The second thing which I observed from the previous point is that once an image had been scanned and an object had been placed on it, I could modify the object to update as I need. I used this property to add one more image to the database and test this theory out, and it worked out. The following video shows this implementation:

2.mp4

3. Although point 2 is not what I really wanted to do, it is a viable alternative. But isn't it really possible to scan an image once again after it has been augmented once?

Thank you so much for your reply. Even though it wasn't exactly what I was looking for, it has given me a different perspective to think about.

[RGregat]

To really understand the issue, I have to try it out myself. Maybe tomorrow. But what I found from the ArCore-Documentation is this

Augmented Image

Once an image is detected, ARCore continues "tracking" the image's position and orientation, even when the image temporarily moves out of the camera view because the user has moved their device. In doing so, ARCore assumes that the image's position and orientation is static, and that the image is not itself moving through the environment.

ArCore keeps the image tracked even if it is not in view. You could try to point for 10-+ seconds somewhere else and eventually the tracking is lost.

You can also check the Tracking-Method of the tracked AugmentedImage

Enable Image Tracking

// Use getTrackingMethod() to determine whether the image is currently
// being tracked by the camera.
when (img.trackingMethod) {
   AugmentedImage.TrackingMethod.LAST_KNOWN_POSE -> {
      // The planar target is currently being tracked based on its last known pose.
   }
   AugmentedImage.TrackingMethod.FULL_TRACKING -> {
     // The planar target is being tracked using the current camera image.
   }
   AugmentedImage.TrackingMethod.NOT_TRACKING -> {
     // The planar target isn't been tracked.
   }
}

The documentation suggest only to place something if the method is FULL_TRACKING. I would say insert some log messages to every Tracking-Method state and see at which situation the state changes.

[sach1n1]

Hallo @RGregat ,

Sorry for the late response.

I also checked the ARCore documentation about Augmented Image page and it says:

ARCore can track up to 20 images simultaneously. ARCore will not simultaneously detect or track multiple instances of the same image.

From this I understand, efforts to rescan an image would be a futile. The suggestion that was made by you to place the robotDetected = false in the handler function helped me to develop my application according to my requirements (update the 3D object placed on an augmented image). However, what I still don't understand is why placing robotDetected = false in the handler function works and my implementation of the same in a different part (commented out).

Thank you so much for helping me out with this issue.