Image Retrieval for Bird Images

[ghost]

Hello, I started an internet recherche to get the images from the scientific bord names. I found out that google changed their REST api access for unlimited image search, so it seems that all solutions based on the shell scripts and python scripts stop working.

[mcguirepr89]

I've stumbled into that same impasse on a previous project, and think we might need to start a hunt for a database solution. Perhaps there are bird photo databases we can import alongside or into the birds database?

[ghost]

I found this:

https://images.search.yahoo.com/search/images?p=SEARCHTERM

[mcguirepr89]

I bet we can work with that -- we'll see how many requests are allowed before it blocks them.

…

On Fri, Oct 15, 2021 at 6:30 PM Christoph Lauer ***@***.***> wrote: I found this: https://images.search.yahoo.com/search/images?p=SEARCHTERM — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOMH3EGCAQI7KPYYPJXDOW3UHCTRDANCNFSM5GC2O26A> .

[ghost]

https://www.bing.com/images/search?q= SEARCHTERM

[ghost]

Searched for: "Zosterops atricapilla"

https://commons.wikimedia.org/w/index.php?search=Zosterops+atricapilla&title=Special:MediaSearch&go=Go

[mcguirepr89]

Yikes! Lots of dead birds!

[ghost]

yes for this species, all other seems to be usable.

[ghost]

Bu ti dont know how to extract the url for the first image. It is always the first image at wikipedia, all species seem to be there. The rest should be really easy, make a folder with the images with the names and link in the php page.

[mcguirepr89]

I should be able to get that -- lynx has a great option to dump urls -- I'm sure that curl could spit them out, too

…

On Fri, Oct 15, 2021 at 7:17 PM Christoph Lauer ***@***.***> wrote: Bu ti dont know how to extract the url for the first image. It is always the first image at wikipedia, all species seem to be there. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#35 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOMH3EBVR3V3HCXNNALB3PLUHCY73ANCNFSM5GC2O26A> .

[DD4WH]

Very nice idea and feature to add photographs!
I think we would have to think about licenses of the photographs before including them in an installation or linking to them. Wikipedia images with a Creatice Commons license should not pose any problems, I think, but other images could be more problematic.

A large image collection can be found here:

https://search.macaulaylibrary.org/catalog

But I am not sure about the licenses for these images . . .

[mcguirepr89]

Unfortunately, from that resource:

The Services are owned by Cornell Lab of Ornithology. All of the content featured or displayed on the Services, including, but not limited 
to, still images, text, pictorial works, video images, still images of video, graphic designs, audio recordings, multimedia combinations, and 
computer programs, including web-based programs (“Content”) is owned or licensed by Cornell Lab of Ornithology. 

[DD4WH]

thats a pity . . .
I guess for educational purposes they could have a special license, but probably its too complicated to apply for that.
Thus the initial idea of using Wikipedia with their CC license is better, I think.

[ghost]

but it has an an easy REST API:
https://search.macaulaylibrary.org/catalog?taxonCode=eurcur&q=SEARCHTERM

Wikipedia is perfect!

[DD4WH]

agreed! And we only need just one nice photograph per species, and not a huge collection. :-)

[ghost]

EXACTELY, i am so happy with the system with works very well. Now I have my new outdoor microphone attached and can see now what flying arround here.

[mcguirepr89]

I'm dumping a complete list of wikimedia commons image links for each species, then I'll see what to do about getting them. My little computer is crippled just running this loop got dump these links! I'll post the links file here when it is finished

[ghost]

:-) you can take simple the first image, I maked some some tests yesterday. I think the first image is the first image on the Wikipedia page for that bird.

[ghost]

The BirdNet demo is maked for a device which has no internet connection (demo truck). But i agree that it is better to have files locally. Hopefully they all fit to GitHub.

[ghost]

Unicode is since 20 years standard. Emojis are now also included. The Icons in the index.html are also Unicode.

[ghost]

I see that on the RaspberryPi the 4GB swap is enabled. Wasn't this a relict from the BirdNet-System and not needed in the Python-Lite version?

[mcguirepr89]

Indeed it was necessary for the BirdNET-system, and I bet we could get away with without using it now. BirdNET-Lite is far less resource intensive and the build process also uses less RAM. We can test easily by disabling the zRAM swapping sudo systemctl disable zram and then rebooting. I will test that out and see how the resources look and will adjust accordingly.

In general, for linux system administration, I tend to stick to the rule that for RAM less than 2GB, you want at least twice your RAM in swap space, and for under 8GB you want your swap to equal your RAM, and over 8GB your swap should be 1/2 your RAM. That may not be totally necessary for this project, but I would feel silly if I didn't prepare the OS to reasonable standards (OS starts with 100M swap, which is often not enough when compiling or building environments). Since the Pi comes in 2GB, 4GB, and 8GB RAM options, I thought 4GB swap fit all criteria nicely. Again, it may not be necessary, so I am happy to change it if it proves to only be a waste of space or can be reduced in size.

[ghost]

ok, is noticed.

[mcguirepr89]

Ok I've got the image links for 93% of the species -- not bad -- that was from an initial scrape only for jpg files -- I can rescrape the missing species for other image files types. I'm going to work on a dynamic script to grab the images as the detections occur, and from a desired list for the BirdNET-Demo feature.

[ghost]

https://www.birds.cornell.edu/home/terms-of-use/

It is allowed to download on a single computer.

1. That you may only download Content that Cornell Lab of Ornithology designates as offered for download to any single computer a single copy of any Content for personal and noncommercial use.

but
2. To not reproduce, distribute, modify, re-post on another site (regardless of the server on which the Content is stored), or sell any Content without specific written authorization from Cornell Lab of Ornithology, unless such use is pursuant to Educational Use.

This means we have to ask first.

[mcguirepr89]

nolinksfound.txt
That is a list of the species for which no jpegs exist in wikimedia commons

[ghost]

Most of them are in Cornwell Library, but not all. But interestingly, most of them are on WIKIPEDIA direct, but not searchable on wiki commons.

[mcguirepr89]

I'm going to work with the 93% of the species list from Wikimedia commons for now, since that will be 5,926 photos

[ghost]

Most of the species from that list are geographically for the South-Eastern-Sea, Africa and South-America. North-America and Europe seems to be photographed.