Q's re hosting and NFT without AR

[Eisenh]

Great work! I love the idea of o lightweight option for NFT.

I have two questions:

1. Would it be possible to host the example, e.g., on github pages, or provide instructions to do so on node? I'm not clever enough with http-server.
2. If not displaying AR, is threejs required at all (e.g. for camera matrix)? Is is possible to extract the quat /corners directly as in AR.js?

An important use case will be to know the location and orientation of the NFT marker (or corners). without displaying AR. I know that isn't the goal, but the jsartoolkitNFT wasm would be perfect. I'm trying to put this into a vue single file component with the source video as a prop.

[kalwalt]

Hi @Eisenh thanks for your interest!

Would it be possible to host the example, e.g., on github pages, or provide instructions to do so on node? I'm not clever enough with http-server.

At the moment this lib is not stable, so i would not host an example, because i think does not make sense yet. And for the other question related about node, i don't think that will work with node as it is...

If not displaying AR, is threejs required at all (e.g. for camera matrix)?

if you want to use artoolkitNFT.min.js or artoolkitNFT_wasm.js three.js is not required at all, you can use Babylon.js liteg.js or other 3D javascript lib. Instead if you want to use arNFT.js is based on three.js.

Is is possible to extract the quat /corners directly as in AR.js?

You are mixing Pattern Markers with NFT markers (image tracking), with Pattern Markers you can retrieve the four corners informations to use in Screen Coordinates directly with the getMarkerInfo function. Unfortunately this is not implemented for Image tracking (NFT). But if you look at my PR kalwalt/jsartoolkit5#42 and in particular this commit kalwalt/jsartoolkit5@b44a407 and the comments/commits after. The problem was that i can't correctly update and fill the position matrix. I think that is possible if i/we can solve that part of the problem...

[Eisenh]

Yes, I did mix up the markers, assuming that the NFT would provide similar location and transform info. I don't know if I can help much. I need a different structure for my projects, where I can pass a canvas to the code for interpretation and return a location of some sort. I tried with OpenCV (wasm, various feature and matching algos) but it was too slow.

[kalwalt]

Which feature and matching algos did you tried?

[Eisenh]

I used SIFT, SURF, ORB and BRIEF and AZAZE using https://github.com/latsic/imgalign. It is quite a useful tool. SIFT descriptors worked best and ORB worked fastest for matching, but I didn't separate the feature step from the matching step yet (Doh!). I'll try to do that first. Feature extraction is obviously the biggest step.

I noticed that in you code the displayed cube slowly moves to the correct location. Is that dampening done in artoolkit.wasm or in three.js?

[kalwalt]

I used SIFT, SURF, ORB and BRIEF and AZAZE using https://github.com/latsic/imgalign. It is quite a useful tool. SIFT descriptors worked best and ORB worked fastest for matching, but I didn't separate the feature step from the matching step yet (Doh!). I'll try to do that first. Feature extraction is obviously the biggest step.

Very interesting indeed, never seen this.

I noticed that in you code the displayed cube slowly moves to the correct location. Is that dampening done in artoolkit.wasm or in three.js?

This is a interpolation routine https://github.com/kalwalt/jsartoolkit5/blob/244b2b23286403e78fa24805b34509dc5a88052f/examples/nft_improved_worker/threejs_worker_gltf.js#L243-L249
done in Three.js to avoid the jttering of the model ( try without the interpolation) This because the trans matrix has a lot of 'noise' to reduce this we added that interp. to reduce, but with much slowness of response. I tried to implement a filter noise ( it came from the Artoolkit lib itsel) but doesn't help so much see this PR kalwalt/jsartoolkit5#36, other people suggested me the Kalman filter, but hadn't so much luck yet...

[kalwalt]

Hi @Eisenh actually i hosted an example https://kalwalt.github.io/jsartoolkitNFT/examples/arNFT_example.html if you want to test. I'm asking also to myself: what difference is between the openCV.js version from https://github.com/latsic/imgalign and that one available at the OpenCV github repository?

[Eisenh]

Thanks. The main differences between the imgalign CV and the full openCV are reduced size (keeping only what is needed for NF detection and matching) and the addition of some custom alignment functions.

[kalwalt]

Thank you for the explanation!

[kalwalt]

@Eisenh if you have not other questions i will close this issue.

[Eisenh]

Thanks for the responses!